@madpilot makes

I have a small home lab that runs Proxmox. It runs my Home Assistant instance, as well as MQTT, dnsmasq and a few other services (including this blog!).

While I back up my working data using kopia and Backblaze, I wanted to set up Proxmox Backup Server so I could recover from any VM level issues quickly.

To do that, I recommissioned an old HP ProLiant Microserver that used to run my lab. It has 4 × 3.5″ HDD drive bays, so can provide plenty of storage, and most importantly - I already had one.

After I set everything up, I was looking at my rack power consumption (I have it attached to a Zigbee smart plug which I monitor via Home Assistant), and noticed that the new server draws around 50W. This is 1.2kWh per day, or 36kWh per month! This is around 5% of my total household consumption.

I work for Amber Electric, an electricity retailer that sells electricity at wholesale rates that fluctuate every 30 minutes, so it’s a bit hard to put an exact monthly price on it. But, we can find the average cost for January 2024 pretty easily from the API and some jq magic:

curl 'https://api.amber.com.au/v1/sites/$SITE_ID/prices?startDate=2024-01-01&endDate=2024-02-01&resolution=30' -H 'accept: application/json' -H 'Authorization: Bearer $AUTH_TOKEN' | jq '[.[] | select(.channelType=="general") | .perKwh] | add/length'

For me, that was 18.34 c/kWh, so $6.60 per month, to have a whole server sitting around not doing much for much of the day.

Now, this isn’t totally fair - I have 6.6kW of solar panels and a 9.6kWh battery, so realistically it actually costs me nothing, BUT! having this server chug 5-6% of my usable battery overnight seems kind of silly, especially since I only run one incremental backup per day.

Automations to the rescue!

My goal for this project was to only run the backup server during the day when the sun was shining.

I needed to solve the following problems:

1. How do I turn the server on?
2. How do I turn the server off?
3. How do I work out if there is enough sun?

How do I turn the server on?

This turned out to be a pretty simple problem to solve, given the work of past Myles who must have set up Wake-on-LAN for this machine. Sending the Magic Packet to the network card happily booted it up.

You can easily test this on OSX by installing wakeonlan from brew. and running

wakeonlan -i $BROADCAST_ADDRESS $MAC_ADDRESS

To figure out the $BROADCAST_ADDRESS go to this Broadcast Address Calculator and enter the IP address of the machine you want to wake up. To find the MAC address, you can run

arp $IP_ADDRESS

For simplicity, the rest of these examples will use a server address of 192.168.0.100 and MAC address of 55:99:30:E6:00:3C

Home assistant has a Wake on Lan integration, so we can set up a switch to turn the machine on! Unfortunately, this integration hasn’t been updated for configuration via the UI yet, so you need to jump in to YAML land.

wake_on_lan:

switch:
  - platform: wake_on_lan
    name: "Proxmox Backup Server"
    mac: 55:99:30:E6:00:3C
    broadcast_address: 192.168.0.255
    host: 192.168.0.100

Restart Home Assistant and you should have a new switch.proxmox_backup_server in your entities. Because we set a host, Home Assistant will ping the host IP to check it’s status, and update the switch accordingly.

So, we can now turn the server on…

How do I turn the server off?

Glad you asked. It turns out, it’s only marginally harder than turning it on.

Proxmox Backup Server shares a bunch of features with Proxmox proper, including an API. The API has an endpoint that allows you to shut down the server via a REST command. There is even a facility to generate an API token, so you don’t need to use your username and password combo.

Click on Configuration > Access Control and click the API Token tab. Then click Add.

Select a user to link to the API key, and give the token a name like home-assistant or similar, then hit Add. Make a copy of the generated key, and save it somewhere safe—it’ll disappear once you close the window.

Next, you need to give the API token permissions to manage power settings. Click on the Permissions tab, and click Add, and then API Token Permission. Select /system/status as the path, as well as the token you just created, and set the Role to Admin.

Now that you have a token, you can use the Home assistant RESTful Command integration to set up an action to switch the server off.

rest_command:
  shutdown_proxmox_backup_server:
    url: "https://192.168.0.100:8007/api2/json/nodes/1/status"
    method: post
    payload: "command=shutdown"
    verify_ssl: false
    content_type: "application/x-www-form-urlencoded"
    headers:
      accept: "application/json, text/html"
      authorization: !secret pbs_api_key

In your secrets.yaml file, you need to prepend the API key you generated with PBSAPIToken=$USERNAME!$TOKEN_NAME:, so if you linked the code to root@pam and named the token homeassistant the YAML entry might look something like this:

pbs_api_key: "PBSAPIToken=root@pam!home_assistant:6af64531-ed7f-47e3-a094-b6e60f965760"

So, now we have a new service rest_command.shutdown_proxmox_backup_server that we can call to switch off the server from Home assistant.

We need to tell the power on lan integration to call that service when the switch is toggled off:

switch:
  - platform: wake_on_lan
    name: "Proxmox Backup Server"
    mac: 55:99:30:E6:00:3C
    broadcast_address: 192.168.0.255
    host: 192.168.0.100
    turn_off:
      service: rest_command.shutdown_proxmox_backup_server

How much sun do we think there will be?

My solar system gets curtailed to match my household load if the wholesale price goes negative, so I can’t really monitor exports to the grid, or solar generation.

I use the Forecast.Solar integration in Home Assistant which will predict my solar generation over the day.

I decided to use this as a trigger for the automation. If the integration is predicting more than 500W of solar, the server should turn on. If there is less, turn it off.

I used this forecast rather than actuals to avoid the server shutting down due to solar curtailment. Is it perfect? No, but it’ll be good enough.

I use a choose condition, so I can keep the whole automation in one file. The trigger is sensor.power_production_now, and the action is if sensor.power_production_now is greater than 500, switch on otherwise switch off. Here is the YAML:

alias: Proxmox Backup Server
description: >-
  Turn on on the Backup server if it’s predicted there is more than 500W of
  solar being generated  
trigger:
  - platform: state
    entity_id:
      - sensor.power_production_now
condition: []
action:
  - choose:
      - conditions:
          - condition: numeric_state
            entity_id: sensor.power_production_now
            above: 500
        sequence:
          - service: switch.turn_on
            metadata: {}
            data: {}
            target:
              entity_id: switch.proxmox_backup_server
    default:
      - service: switch.turn_off
        metadata: {}
        data: {}
        target:
          entity_id: switch.proxmox_backup_server
mode: single

Tell Proxmox to run backups during the day

The last thing to set is to kick off Proxmox backups at midday, rather than at 2am - no point trying to connect to a server that is currently sleeping.

In the Proxmox console, Go to Datacenter > Backups and edit all the schedules to read 12:00.

And that’s it! You can see this history snapshot of when the server is on and off.

After cancelling the order that was clearly never going to arrive, an order from a new supplier has completed the y-axis for my homebuilt CNC mill.

I used left over aluminium to create four adapters to hold the base board to the slides. I’m using a 200x180mm extrusion as it gives me a bunch of work piece holding options.

Once I assembled everything, I did notice a couple of issues.

The spindle isn’t centered

The spindle is about 20mm off centre in the Y direction, so I needed to adjust the position of the front brackets to stop them falling off when the y-axis was at full stroke.

This also meant I needed to move the end stop 20mm forward so the home position was in the right place.

Spindle travel

The spindle was too high. I was trying to avoid having to cut the vertical structs, but removing 40mm from all four seemed the easiest solution.

This does make using a vice a little less convenient, but the main use case for this mill will be PCBs so being able to get close to the spoil board is more important.

The lead screw saga

This issue was more annoying: the Y-lead screw was now too short. Completely my fault – this was 100% a design issue. At this point in the project I’d been riffing without CADing things first and didn’t realise that the position I had the anti-backlash nut lost 40mm of travel. Add another 20mm because of the spindle offset and I was 60mm out.

I discovered this by witnessing the anti backlash nut spring flying across the room as it spun off the edge of the screw.

So I had a conundrum. One of the design goals of this project was to reuse as much if the Makibox as possible. I had already replaced all the 4mm linear rods with 12mm linear rail, leaving just the steppers, lead screws and controller board.

I could either redesign the y-axis, which would require a bunch of replacement parts, waiting for said parts and time; or I could just buy a longer lead screw.

I found a 300mm T8x8 on EBay in Australia for $30 delivered and it could arrive in three days. I did this.

Alignment – Y axis

At this point I stripped everything back so I could align all the pieces.

I found a spare bathroom tile that I planned on using as a ground plane – it seemed flat enough for my purposes.

I started by squaring up the base frame using an engineer’s square.

Next a place one of the y-axis rails loosely in place using a 3D printed spacer. I dury rigged my depth gauge so I could dial it in – I got it to with in 0.05mm. Machinists would baulk at that but I’m fine with it.

I cheated a bit in the second rail: I simply put the bed on and rolled it up and down a few times to get alignment.

Finally I used the depth gauge to get the bed square.

The version of the Makibox that I purchased came with a printrboard, however I fried that years ago. Makibox shipped me a replacement Print5D D8 controller.

I knew very little about this board, and I wondered how I was going to adapt it for CNC use.

I had some questions:

1. How do I send it G-Code? Do I need to use the software that came with the Makibox?
2. Controlling the three axes and endstops was easy, but how would I run the spindle?
3. After more research about milling PCBs, I found out that using a bed levelling algorithm would give the best results – could the board do that?

After some googling, I found that a kind user had archived all of the original Makibox forums. A post about the D8 firmware lead me to the Makible bitbucket account, which not only housed the original firmware source, it also had the schematics!

Some more random googling yielded another interesting tidbit of information: Marlin has hardware definitions for the D8 board.

The original firmware (from 2014) was based on Sprinter which was forked and became Marlin.

Fun fact: Marlin is also what my Lulzbot Mini uses.

Sending G-Code

I know had an answer to question 1: Marlin appears as a serial device that you can simply send G-Code to. If you have screen installed on your *nix like computer you can just connect (by passing the path to your TTY device) and directly enter G-Codes – although using a G-Code sender is easier.

After wiring up the endstops and motors (only the X and Z because I’m still waiting on the slides for the Y) I entered G28 which runs the homing routine. It worked!

The next problem to solve was driving the spindle. The Makibox schematic revealed the drive circuit for both the hotbed and hotend use a MOSFET to switch a 24V line (using PWM). The MOSFET is rated to 40A, which is plenty to run the 5A spindle I have.

Welcome to 2018

This current firmware is ancient, is tuned to the Makibox build area and doesn’t support autolevelling or controlling a spindle, so I needed to upgrade to the latest Marlin. This was far easier than anticipated.

The board is based on the printrboard which is based on the teensy 2.0++, so the first step is to load the teensy board definitions into the Arduino IDE. Unfortunately, this isn’t via the usual method of adding a URL to the board manager – there is a installable which manually handles installation, and adds a bunch of wizards that, if you have done any Arduino development, kind of get in the way.

Regardless, it allowed me to choose the teensy 2.0+ as a build target.

Next, I modified the configuration.h and configuration_adv.h file with the new build area parameters, and set the various options. A number of these settings hadn’t changed since Sprinter, and I was able to just copy the values. I did have to modify a non-configuration file to allow pin 15 to be defined as a PWM pin.

After compiling, I put the board in bootloader mode by placing a jumper over the boot pins and clicking reset.

Once in bootloader mode I erased the EEPROM and uploaded the binary hex file via dfl-programmer.

sudo dfu-programmer at90usb1286 erase
sudo dfu-programmer at90usb1286 flash Marlin.hex

The spindle parameters took a little bit of trial and error to get right:

To calculate the SPEED_POWER_SLOPE use the formula:

(SPEED_POWER_MAX - SPEED_POWER_MIN) / 255

I decided to use 0%-100% as my MIN and MAX, so the value I needed was (100 – 0) / 255 = .392156863.

Using that range allows me to use different gear ratios and to drive a laser cutter without modifying the firmware.

I have purchased a copy of GWizard which calculates spindle speeds and feed rates, and it will output a power percentage based on the profile of the spindle, which makes the calculation easy.

I’m not entirely sure what SPEED_POWER_INTERCEPT is, but I’d guess that it is a way of offsetting the speed. I’m not sure why you would need to do that.

The new firmware has given me the ability to set the spindle speed via M3, M4 and M5 (M3 and M4 do the same thing because the hardware can’t reverse the direction of the spindle), and in theory perform an auto-level – I say in theory, because I don’t have a levelling probe, nor do I have the Y-axis linear slides. I have now followed up with the AliExpress seller to see where they are.

So once again, I’m blocked – but progress has been made!

In my previous post, I realised that the 4mm steel rods weren’t going to cut it when milling – they had way too much play, so I promptly ordered two 200mm linear rails (for the X-axis), two 400mm linear rails (for the Y-axis) and two 150mm 12mm linear rods and bearings (Z-axis).

The 200mm slides and the 150mm rods arrived promptly, but there was a delay on the 400mm slides. That, plus me needing a fair chunk of time to measure, drill and tap a bunch of holes meant I parked the project for a couple of weeks.

I finally managed to get a couple of days off while I changed jobs, so I arranged a hot date with my drill press.

The Tappening

The x-axis slide has four mounting screws which required tapped holes to be drilled into the two horizontal 4040 aluminium extrusions. Now, I’ve not done much in the way of metalwork since year 11 (some 18 years ago) and it took me ages to mark, drill and tap the holes. This was a good warm up for Z-axis face plate.

The face place required a total of 20 holes: four 2.5mm tapped to an M3 and sixteen 5.5mm counterbored to 10mm. It’s not the neatest job in the world, but it is functional. And I only misaligned one hole, which am actually amazed by.

I have stuck with 3D printed anti-backlash nut holders, with the intention of milling them out of aluminium once the mill is completed. This is why they are the shape they are – so they will be easy to mill.

The Z-axis

The same applies to the Z-axis end caps – I added grub screws to hold the linear rail which adds some rigidity – and they will be especially helpful on the milled version.

After doing a fit test it became clear that the motor couplers were too springy (the ones I bought are designed to deal with things being out of alignment) so I’ve now ordered some rigid ones that will eliminate any slop.

I 3D printed a Z-axis carriage to test fit, but it became clear it was too big (90mm tall) which meant my Z-axis would be less than my 70mm design goal. I’ve simplified it to be more box like (easier to mill down the track), and reduced the height to 60mm, by cutting down the flange on the anti-backlash nut, and by making it a two piece part. At this point, I calculate a 78mm of travel in the Z direction.

I now need to fabricate the aluminium face part part, which the carriage and the spindle screw in to.

One of my other hobbies is food. I do a lot of cooking, with a particular interest in slow cooking: I regularly fire up my smoker and cook a pork shoulder or brisket, and I’ve started learning the finer points of sous vide cooking. One thing I have been interested in is curing meat.

My mates Matt and Lachlan also expressed an interest, so we booked to a course at The Artisan’s Bottega. During the course the instructor explained to us that there was only a couple of weeks of meat curing weather left – they need to be done before it got too hot. None of us wanted to wait a whole year, so we – with help from some other friends: Remo and Dan – decided to do our first ever batch the weekend after.

Ordering the meat

To keep things manageable, we ordered 12kg of pork shoulder from Paddock to Table in Laverton (we also ordered 3kg of pork belly to make bacon). Luckily the butcher was himself an experienced salami maker, and knew exactly what we needed. Since I have a Kitchenaid with a mincing and sausage attachment, we opted for it to be boned, but not minced.

Having it boned cost a couple of bucks a kilo more, but meant we didn’t have to do it – saving us a bunch of time. It’s better to mince it fresh though, as minced meat has a much bigger surface area than unminced meat, which bacteria loves. Something that bacteria doesn’t love is curing salt, so we quickly minced the meat, got it salted and flavoured.

Flavours

We did two batches: a very traditional one, and a second moroccan inspired one (global head of flavour, Lachlan was freestyling).

Salami 1

• 5kg pork + fat
• 8 cloves garlic
• 5 tbs smokey paprika
• 5 tbs chilli flakes
• 1.5 tbs ground fennel seeds
• 0.5 tbs garlic powder
• 1tbs ground cumin
• 150g passata
• 110g flaked salt
• 2 cups Sangiovese

Salami 2

• 5kg pork + fat
• 2 tbs ground cumin
• 2 tbs ground coriander seed
• 2 tbs Hungarian paprika
• 1 tsp cayenne pepper
• 1 tsp ground cloves
• 1 tsp cracked black pepper
• 150g passata
• 110g table salt
• 1 cup Sangiovese

We were advised to cook up a little bit to test for flavours (which is what you do when you make fresh sausages), but to bear in mind that the sample will taste saltier than the final product (more on this later).

Lunch

The salami kits we got given during the course had enough stuff to make 10kg, so we turned the remaining 2kg of meat into fresh sausages. This turned out to be an excellent idea: not only did it use up the remaining sausage cases I had, it provided us all with a delicious lunch.

Prepping the casings

The kits we had gave us synthetic casings, which we needed to be soaked in water before use – we threw them in a bowl
of water for 30 minutes while we prepped the mince.

Equipment

We used the following:

1. Kitchenaid with the meat grinder and sausage attachment. Use the coarse plate (the one with the bigger holes)
2. Two plastic tubs for mixing the meat
3. An old tennis ball tube and beer bottle to help get the netting on.
4. A metal bowl for soaking the casings
5. Food safe Latex gloves

Cleaning

Bacteria loves raw meat, so we cleaned all of our equipment using a 1:1 mixture of white vinegar and water.

You can apparently also get food safe cleaning alcohol, but we used what we had.

Prepping the mince

Using the coarse grinding attachment on the Kitchenaid we ground the meat into plastic trays – 5kg in to each.

Next, we sprinkled the regular salt and curing salt, poured over the sauce and combined. You really need to combine the meat well – the mixture should be quite dense. Use a grabbing action, and don’t forget to scoop up the meat from the bottom of the tray to make sure it all gets a good massage.

Our instructor gave us a handy hint: if you grab a handful of meat and hold it upside down, it should stick to your hand.

Stuffing the casings

Using the sausage attachment on the Kitchenaid, we filled each casing. This is a pretty crucial step to get right, as there can be no air with in the sausage – air breeds bacteria. While the sausage attachment did an ok job, there are dedicated sausage stuffers that looks like a massive syringe that will do a much better job.

Tying off

The final step is to tie off each sausage. As we were using synthetic casings, one end was already closed, so we only needed to do one knot. We first pinched the end to get out as much air as possible, then twisted to give us a spot to tie the butcher’s string.

The knot will be used to hang the salami, so make sure they are tight – we used an under and over technique, so there was around three single knots, finishing with a final double knot. Leave enough string to create a loop.

Give the end of the casing a little rinse in water to get rid of any meat that may have lodged there.

To get out any extra air, we also tied a string around the middle of the salami and pulled it tight (the synthetic casings are pretty strong – I imagine the natural casings wouldn’t take as much pressure). Tie the string off.

Netting

The final step before hanging is to put the salami in netting. This helps keep the pressure on the salami as they dry and shrink. The netting is quite difficult to stretch, so putting it on without the assistance of a tool is really difficult. There are plastic tools you can buy to help, but we didn’t have one, so we improvised with a tennis ball can and a long neck beer bottle.

We took the end caps off the tube, placed the base of the beer bottle at one end and slowly pushed up the neck of the bottle, past the body and on to the tube – leave a bit of netting hanging off the end.

Drop each salami through the tube – it should catch the netting that is hanging, allowing you to pull it through, encasing the salami in the netting.

Cut the netting at the end – be careful not to cut the string!

Repeat for all the other salamis

Hanging

I have an enclosed garage, which I don’t park a car in, so we hung them up there on a piece of wire.

We were told that it should take about six-weeks.

Things to look for

Of course, like most cooking, time is just a easy proxy for doneness – having actual indicators is a much better system,

Weight: During the two weeks the salami lose a lot of moisture and should drop around 30% of their weight and visibly shrink. If that doesn’t happen you may have air in the centre which has rotted the meat.

Hardness: the loss of moisture also causes the salami to get hard.

At the end of the six weeks they had both shrunk and got really hard, so I pulled them down and cut them open.

The opening

I sliced each open either side of the middle tie, and cut the ends off. I was looking for a nice dark red colour – brown or grey is bad.

I had a suspicion that the casings has dried out – a problem caused by low humidity, and cutting them open confirmed that. Some of the casings had lifted from the meat at the end, and the meat had gone off at these points. So I discarded those sections. All in all we lost about two salami worth – not a bad yield for first timers.

Taste test

The traditional recipe was good, but nothing spectacular – it probably had a little too much fennel. The Moroccan inspired recipe was delicious with a nice heat to it. Both were possibly a little too salty.

Things to fix

Avoid spoilage – We need a proper sausage stuffer. The sausage attachment on the Kitchenaid is not ideal, and curing meats is way more prone to bacteria infection than fresh sausages that are cooked with in a couple of days.

Dry casings – there is possibly two issues here:

1. More humidity. Looking at my graphs my garage humidity was too low: around 50-60%. I’m looking into building a curing chamber so I can control that (and the temperature) with science.
2. Using synthetic casings – they are more plasticity than natural ones.
   I suspect the humidity is the main culprit here, so I’ll stick with the casings for the next batch, so as to not change too many dependent variables.

Improve efficiency – I would also like to get a clamping gun to ensure all the ends are properly sealed, which should further reduce spoilage. I would also look into getting a netting tool, because trying to get the nets on is a pain without one.

Re-designing the X-axis of my Makibox to PCB mill conversion has been a challenge in constraints. Due to an ordering mishap, and lack of engineering designs for the spindle I have been racking my brain to re-use the parts I have to create a workable X-axis.

The three constraints I have to work with:

• The spindle can’t be too low – I want at least 70mm of vertical travel
• The spindle can’t be too high – The further the cutting tool is from the centre of the holder, the more torque it needs to deal with
• The spindle can’t be too far forward – I’d like the cutting tool to sit as close to the centre of the frame, so it can reach the full range of motion. The shaft of the spindle sits around 40mm from it’s mounting plate, which puts it out nearly 80mm when you include the Z-axis linear driver.

I looked at mounting the stepper motor on top of the vertical extrusion, but it failed constraint 3. It also gave me no where to mount the top steel rod. I toyed around with moving the steel rods but centred either side of the lead screw seems the best place for them.

I settled on a design where the motor hangs off the side of the vertical extrusion, with the steel rods mounted either side.

The test build

After printing motor and bearing holders, and the carriage I put everything together. I held the spindle against the carriage to get an idea of placement, and it became clear that the 4mm steel rods were going to flex too much under load.

They would be fine for a 3D printer or as a laser cutter, but even the small forces from a PCB mill bit would cause issues. It was time to replace the rods with something more robust.

I jumped on Aliexpress and found these 200mm linear guides that have a 12mm steel rod, and support for the full length, so bending should not be an issue.

I also decided to replace the Y-axis rods with the 400mm version of the linear guide. This will remove the flex in the bed and will make alignment easier (the three points of contact design I used was difficult to square).

While I was at it I ordered some linear bearings for the Z-axis for good measure.

A substantial redesign.

I have one 205mm cross bar, which a perfect support for one of the linear rails – I’ll need another one. I’ll also need another two, taller vertical extrusions to hold the cross bars. The two shorter vertical extrusions will hold the motor.

Here is a rough render without any joining hardware so you get the idea.

I’ve had to move the vertical extrusions to the rear of the frame (which is fine, there are still two connection points there), giving the spindle plenty of room.

I’m going to design and print some new motor and bearing mounts for the X and Y axis. They will be much simpler as they no longer need to support the steel rods.

The rear support for the Z-axis will be 10mm aluminium, as I can get the supplier to cut it to the required 138mm length. The end caps – which act as a motor holder and rod support – will be plastic to start (I don’t have facilities to mill aluminium… yet). My intention is to mill replacements once the machine is complete.

I have been doing a bunch of user interviews at work. It’s been difficult to get our users in front of a computer, or to get them to install video conferencing software, so I’ve been calling them on the telephone. I find that taking notes while I interview people kills my flow, and I’m really not very good at it, so I needed a way to easily record these phone calls, and then get them transcribed.

There are a bunch of solutions out there, but they all seem to rely on a third party app that make a VoIP call. This presented me with three problems:

1. Our spotty office wifi caused drop outs – and when it did work, the quality was terrible (thanks Australia!);
2. The incoming number was a either blocked or some weird overseas number, which users promptly ignored
3. Automatic transcription of phone calls is woeful – because of the low bandwidth, the audio signal isn’t great, and computers do a really bad job at translating it to text.

When I was last at JSConf, I got chatting to Phil Nash who is a developer evangelist at Twilio. I asked him whether I could setup a number that I could call, have me enter a number, dial that number and record the call. He said it should be easy.

Challenge accepted.

Spoiler alert: It is.

Note: This code and deployment process isn’t actually the one I used at work. We use Golang, which is way more lines of code, and has way more boiler plate – and I needed to write an abstraction around TwiML – so I chose to rewrite it in Python here for simplicity’s sake. We also use CloudFormation to create Lambdas and API gateways, and have a CI build pipeline to deploy them, which isn’t conducive to a pithy blog post. Does this process work? Yes. Would you use it in real world in a production environment? Up to you. 🤷‍♀️

An overview

At a high level, this is what happens when you dial the number and request an outgoing call:

1. Twilio answers your call
2. Twilio makes a request to an Amazon Lambda via an API Gateway, which returns some TwiML instructing Twilio’s voice engine to ask you to enter a phone number.
3. It waits for you to enter the phone number followed by the hash key
4. When it detected the hash key, it makes another request to the lambda, this time with the number you entered. The lambda returns an instruction to dial the number (after some normalisation – more on this later), join the calls, and to record both sides of the conversation.

After this happens, you can log in to the Twilio console, download the MP3, and upload that to Rev.com where real-life humans transcribe the conversation.

The code

from twilio.twiml.voice_response import VoiceResponse, Say, Gather, Dial
from urllib.parse import parse_qs

LANGUAGE="en-AU"
AUTHORISED_NUMBERS = ['ADD YOUR PHONE NUMBER HERE IN INTERNATIONAL FORMAT ie +61400000000']

def authorized_number(number):
    return number in AUTHORISED_NUMBERS

def say(words):
    return Say(words, voice="alice", language=LANGUAGE)

def get_outgoing_number(request):
    response = VoiceResponse()

    action = "/" + request['requestContext']['stage'] + "/ivr"
    words = say("Please dial the number you would like to call, followed by the hash key.")

    gather = Gather(action=action)
    gather.append(words)
    response.append(gather)

    return response

def add_country_code(number):
    if number[0] == "+":
        return number
    elif number[0] == "0":
        return "+61" + number[1:]
    elif number[0:2] == "13":
        return "+61" + number
    elif number[0:2] == "1800":
        return "+61" + number

def hangup():
    response = VoiceResponse()
    response.hangup()
    return response

def handle_ivr_input(params):
    to = params['Digits'][0]
    dial = Dial(add_country_code(to), record="record-from-answer-dual", caller_id=params['Caller'])

    response = VoiceResponse()
    response.append(say("Calling."))
    response.append(dial)
    return response

def handler(request, context):
    path = request['path']
    params = parse_qs(request['body'])

    response = ""
    if path == "/incoming":
        if authorized_number(params["Caller"][0]):
            response = get_outgoing_number(request)
        else:
            response = hangup()
    elif path == "/ivr":
        response = handle_ivr_input(params)
    else:
        return {
            'body': "Action not defined",
            'statusCode': 404,
            'isBase64Encoded': False,
            'headers': { 'context_type': 'text/plain' }
        }

    return {
        'body': str(response),
        'statusCode': 200,
        'isBase64Encoded': False,
        'headers': { 'content_type': 'text/xml' }
    }

Code is also on GitHub.

I’m in Australia, so there is a little bit of localization happing here: I set the voice of Alice (the most human sounding robot that Twilio has) to Australian, and I insert the Australian country code if it is not already set. Twilio doesn’t do this automatically, and it’s a pain to replace the first 0 with a +61 for every call.

When the call is made, the caller ID is set to the number you called from, so the call looks like it came from you. You need to authorise Twilio to do that.

I’ve included a hard-coded allow-list (AUTHORISED_NUMBERS) of phone numbers who can make outgoing phone calls. If a number that is not on the list tries to call the number, they just get hung up on. You wouldn’t want someone accidentally stumbling across the number and racking up phone bills. I guess at least you would have recordings as evidence…

Note: the code doesn’t know about area codes, so if you are calling land lines (Wikipedia entry if you are under 30 and don’t know what they are) you will always need to include your area code.

Cool. So what do you do with this code? It needs packaging and uploading to Amazon.

Packaging the script

(Want to skip this bit? Here is the ZIP – you can edit it in the Lambda console once you upload)

You will need Python 3.6 for this. Instructions for OSX, Linux and Windows. Good Luck.

git clone git https://github.com/madpilot/twilio-call-recorder
cd twilio-call-recorder
pip install twilio -t ./
find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf
zip -r lambda.zip *

Creating the Lambda

1. Log in to your AWS account, and go to https://console.aws.amazon.com/lambda/home
2. Click Create function
3. Then Author from scratch
4. Name your lambda: twilioCallRecorder
5. Select Python 3.6 as the runtime
6. Select the Create new role from template(s) option
7. Name the role: twilioCallRecorderRole
8. Click Create Function

Your screen should look similar to this:

Important! Don’t Add a trigger from this screen! We need to set the API gateway in a non-standard way, and it just means you’ll have to delete the one this page sets up.

Uploading the Code

1. Upload ZIP File
2. Set the handler to recorder.handler
3. Click Save
4. In the code editor, Add all the phone numbers who can make outgoing calls to the AUTHORISED_NUMBERS list (include +61 at the beginning)
5. Click Test
6. Set the event name to Incoming”

Set the payload to

{
  "path": "/incoming",
  "body": "Caller=%2B61400000000",
  "requestContext": {
    "stage": "default"
  }
}

• Click Create
• Click Test

Setting up the Web API Gateway

1. Click Get Started
2. Click New API
3. Name the API Twilio Call Recorder
4. Click Create API
5. From the Actions menu, select Create Resource
6. Check the Configure as Proxy Resource option
7. Click Create Resource
8. Start typing the name you gave the Lambda (twilioCallRecorder) – it should suggest the full name
9. Click Save
10. Click Ok
11. Click Test
12. Select Post in the Method drop down
13. Select /incoming as path
14. Set RequestBody to

Caller=%2B61400000000

replacing 61400000000 with one of the numbers in your allow list

15. Click Test

If that all worked, you should see a success message.

Deploy the API Gateway

1. From the Actions menu, select Deploy API
2. Enter Default as the name
3. Click Deploy

Copy the invoke URL. In a terminal (if you are on a Mac):

curl `pbpaste`/incoming -d "Caller=%2B61400000000"

Congratulations! You have setup the API Gateway and Lambda correctly!

Setup Twilio

See the documentation around webhooks on the Twilio website – paste in the URL from the API gateway, and you are good to go.

Making a call

The part you have been waiting for! Pick up your phone, and dial the incoming number you setup at Twilio. If all is well, you should hear a lovely woman’s voice asking you the number you wish to dial. Enter the number, and hit the hash key. After a moment, the call will be connected!

Once you hang up, you can log in to the Twilio console, browse to the Programmable Voice section and click the Call log. You should see the call you made. Click on it and you will be able to download a WAV or MP3 version of the recording.

Now, you just need to download it (I chose the MP3, because it will be faster), and upload it to Rev.com. After a couple of hours, you will have a high quality transcription of your conversation. It’s really very easy!

I’ve now built the aluminium frame and completed the Y-axis. Of course, this was not without it’s problems.

Firstly, I misordered – I was one set of uprights short. This may not be a massive problem though, as I hadn’t taken into account the size of the spindle when designing the X-axis, and it wouldn’t have fitted in the configuration I had designed for – That’s what you get for forging ahead with out good engineering drawings.

Other minor issues were easily fixed by reprinting some parts – I added limit switches into the motor and bearing holders (though I haven’t worked out how to make the switch on the bearing holder work yet), I made the driver carriage wider so the whole anti-backlash nut fitted completely and I changed the shape of the passenger carriage so it could slide over the entire stroke.

Everything went together quite well – I did my best to square everything up, using a shim I printed – the holders may not be exactly in the middle, but they are all consistently out, which is the main thing.

I was a little concerned that both carriages were rotating around the z-axis, but realised that was because they weren’t joined yet, so there was only two points of contact, rather than four.

To fix that, I cut out a 205mm x 205mm MDF spoil board, and attached it using a 0.33mm feeler gauge squared it against one of the uprights.

I’m not sure I can do more alignment without having a X-axis, which requires a redesign.
See the video below for the test! (Excuse the upright video – I need to get an iPhone holder)

Hugo just turned one, and to celebrate we bought him a Duktig kitchen from IKEA. Like everyone else we painted and added fake subway tiles, but I wanted to take it a bit further and add a working microwave panel.

Hugo is obsessed with pressing buttons and turning knobs – We were staying at an AirBNB recently that had a microwave that you controlled using one big knob and he loved it – so I thought I would model that.

I ordered a 4-digit 7 segment display, a rotary encoder and an aluminium knob, which I planned to run off an Arduino Nano.

Rotary Encoders

A rotary encoder looks a lot like a potentiometer, however they work quite differently. For starters, there is no limiter so they rotate through a full 360 degrees.

But the main difference is how you interface with them: they output a digital code called a Gray code. Named after Frank Gray, it ensures that one-bit only ever changes at one time so we can be sure of the direction the shaft turns.

Of course, the library we are using hides this from us, so all we care about is the value that comes back.

The encoder I’m using also has a push button, which I’ll use to start and stop the timer. The library also abstracts this.

The display

The four-digit seven-segment display has a built in driver chip that means we only need two pins to drive it: a clock and data. Again, there is a convenient library that deals with pushing data to the board.

The LEDs

I had six yellow LEDs that I pulled out of the night light I gutted to make an interim night light, so I repurposed them for this project. As the draw a total of about 120mA when on, they can’t be driven by the Arduino directly – I’m using a BC547 transistor to drive them.

The speaker

I added a cheap, 8 ohm streaker than I’m driving directly off one of the Arduino digital outputs. There is a tone library that turns a GPIO on and off quickly enough to create a tone. I maybe regretting this already though – it can get quite annoying.

Sleeping

This microwave controller needs to be battery operated. To do that without costing me a small fortune in batteries, the controller needs to turn itself off. Luckily, the Arduino Nano supports a deep sleep which reduces it’s current draw. The Nano can then be woken up by a pulse on either pin 2 or pin 3.

By feeding the signal from the rotary encoder and the switch into those pins, any rotation of button pressing will wake up the processor.

There is a slight problem though: there are two pins on the rotary encoder, and one on the switch: we need another input.

Or do we?

Rotary encoders have a physical “click” (called a segment). On the particular encoder I have, each click goes through a complete Gray code cycle, meaning there is a guaranteed two logic level transitions on each pin per segment. And, if you have ever watched a small child spin a knob, you would have noticed that their developing fine motor skills result in big rotations. In this case I can happily take the output of one of the encoder pins, and be confident that it will always wake.

So, the switch is wired to GPIO 2, encoder pin A is wired to GPIO 3 and pin B is wired to GPIO 4.

The code

There are four main parts of the code: reading the encoder, the display code, the code that puts the MCU to sleep, and the interupt handler that wakes it back up again.

Most of the heavy lifting for interfacing with the encoder and display is done by two libraries.

The encoder library uses polling rather than hardware interrupts (which is good because I would have run out of interruptible pins). Every millisecond it checks the state of the encoder, and can work out if it has been spun or not. This library exposes the delta, we we use to increment or decrement the display.

The library also provides a callback that is fired on button clicks. Each clock toggles the running variable and the returns control back to the loop.

The display library at it’s lower levels takes four bytes – one for each number. Each bit of each byte represents a segment of the display. Thankfully there are some helper methods to display common things like numbers.

You can download the code on Github.

The loop

Because all of the inputs are handled via interupts, the loop just needs to deal with updating the display.

The left variable represents the number of seconds remaining. The first thing to do is adjust it if the encoder changed. We simply add the delta to it. Note that his happens regardless of whether the timer is running or not – this means you can add or remove time even when the microwave is running.

If the timer is running and the LED is off, I turn it on (and vis-e-versa).

If the timer is running the variable automatically gets decremented every 1000 milliseconds.

If the variable is 0, the timer stops and I display the word end and sound the beeper. (The LEDs will also go off)

The display routine gets called every 500ms. Why every half second? So we can flash the dot to show that the timer is running!

The build

I was hoping this would be a quick weekend project, so I threw the circuit together using perf board. Of course everything always takes longer than you expect, but it kind of worked out well, because the perf board made the feature creep easier to deal with.

I 3D printed a “case” though my printer’s build area was too small to cover the whole area, so I made an MDF backing plate. Pro tip: don’t use pin to mark construction lines: paint won’t stick to it. Also: prime MDF – it absorbs paint like no body’s business. What should have been two coats required four.

Power

During testing I realised that there were LEDs on both the Arduino and the display that stayed on regardless of whether the Arduino was asleep or not. I measure the power draw whilst sleeping at it was sitting at about 15mA. A AA battery has a capacity of around 2400mAH, so a set would last around 160 hours or 6 days.

Clearly that is too high.

I removed the LEDs and that brought the draw down to 2mA, which buys me 1200 hours or 50 days. Better, but not amazing. Unfortunately, the Nano isn’t great as a lower power device. If I could be bothered removing the chip from the perf board, I could remove the FTDI drive and power regulator and probably claw back a bit of quiescent current.

I have ordered a boost converter so I can run the board off two batteries at 95% efficiency rather than the four at the moment so I might just use a couple of C cells which have 8,000 mAH capacity (nearly half a year).

See the microwave in action!

I’m pretty happy with how the aluminium frame has come together. I’ve kept the X-axis pretty simple, though I tried a few iterations before coming to this shape.

Originally I had the vertical X-axis supports butted on the top of the horizontal Y axis base, but I was concerned with keeping the vertical… vertical. I could have used right angle brackets, but decided that by putting them in the inside of the base, I can add additional points of contact that would better support them. This configuration also gives a slightly larger base, so should make it slightly more stable. They are now attached in two dimensions which effectively works like a right-angle bracket.

The motor mount and bearing mounts are pretty simple, bridging the two verticals on each side. I toyed with running the bolts in a horizontal extrusion that would stop them from slipping down, but that would stop me from adjusting the heights when tramming.

On the topic of adjustments, I’m starting to regret the dual-carriage design, as there is an extra dimension that needs to be aligned – not only do I need to align the X, Y and Z axes, I need to make sure both slides are exactly parallel. I’ve bought a digital dial indicator which should help in squaring everything up – we’ll see how that goes.

I now have a complete bill of materials for the frame:

The order cost me just shy of $200, which already takes me over my $250 budget, but I’ve been reconsidering how much I’m spending, as I assumed I could use my Dremel 4300. After a bunch of reading, I think I’ll need to by a different spindle that has less runout.