@madpilot rants

Setting up Buildbox when you run your own Git server

I’ve been trying to get a CI server for 88 Miles sorted out for ages. I tried to cobble something together before, but lost interest trying to keep track of previous builds and Ruby environments and other stuff. Me being me, I run my own Git server, so many of the existing CI servers out there won’t work for me (They assume GitHub), and I don’t really feel comfortable sending out source on a private project to a third party server. I also have a VM in my office that runs various dev machines, so I have the hardware to do it. Well, it turns out that a buddy of mine has been building a CI server that is a perfect fit for my purposes! It’s called Buildbox, and it comprises of an agent that you run on your hardware that manages builds for you. This is roughly what I did to get it running.

The Build Server

I setup a VM running Gentoo, with 1Gb of RAM and two virtual CPUs. It is as close to my prod environment as I can get it, running RVM – this is important, as there is a bit of a trick to this. I also set up a specific user (called build), that will do the work. I initiate the Buildbox daemon using monit, so if it ever dies, it should get restarted automatically.

Note: I’ve omitted the Buildbox setup instructions – The website does a better job than I could.

The wrapper script

Because I’m running RVM, and because monit is fairly dumb in setting up bash environments, I wrote a small wrapper script that will start and stop the Buildbox executable:

#!/bin/bash
case $1 in
        start)
                if [[ -s "$HOME/.rvm/scripts/rvm" ]]; then
                        source "$HOME/.rvm/scripts/rvm"
                elif [[ -s "/usr/local/rvm/scripts/rvm" ]]; then
                        source "/usr/local/rvm/scripts/rvm"
                else
                        printf "ERROR: An RVM installation was not found.\n"
                        exit -1
                fi

                rvm use ruby-2.0.0-p247
                buildbox agent:start &
                echo $! > /var/run/buildbox/buildbox.pid
                ;;
        stop)
                kill `cat /var/run/buildbox/buildbox.pid`
                rm /var/run/buildbox/buildbox.pid
                ;;
        *)
                echo "Usage: buildbox-wrapper {start|stop}";;
esac
exit 0

If you call buildbox-wrapper start it sets up a RVM environment, then runs the buildbox agent and then saves the PID to /var/run/buildbox/buildbox.pid (Make sure the /var/run/buildbox directory is writable by the build user). Calling buildbox-wrapper stop reads the pid file and kills the agent.

The monit config

Add the following to your monitrc:

check process buildbox with pidfile /var/run/buildbox/buildbox.pid
        start program = "/bin/su - build /bin/bash --login -c '/home/build/scripts/buildbox-wrapper start'"
        stop program = "/bin/su - build /bin/bash --login -c '/home/build/scripts/buildbox-wrapper stop'"

This will change to the build user, then run the wrapper script.

My build script

You enter this into the code section of the web UI (I was a little confused by this – I didn’t realise it was editable!)

#!/bin/bash
set -e
set -x

if [ ! -d ".git" ]; then
  git clone "$BUILDBOX_REPO" . -q
fi

git clean -fd
git fetch -q
git checkout -qf "$BUILDBOX_COMMIT"

bundle install
bundle exec rake db:schema:load
bundle exec rake minitest:all

This is pretty much a cut and paste from the Buildbox website, except I run Minitest. I also tmp/screenshots/**/* and coverage/**/* to the artifacts section. Artifacts are get uploaded after a build is complete. I use them to upload my screenshots from all of my integration tests, as well as my coverage reports.

Git post-receive

This script belongs on your Git server in the hooks directory. Name is post-receive

#!/bin/bash

while read oldrev newrev ref
do
  branch=`echo $ref | cut -d/ -f3`
  author=`git log -1 HEAD --format="format:%an"`
  email=`git log -1 HEAD --format="format:%ae"`
  message=`git log -1 HEAD --format="format:%B"`

  echo "Pushing to Buildbox..."

  curl "https://api.buildbox.io/v1/projects/[username]/[projectname]/builds?api_key=[apikey]" \
          -s \
          -X POST \
          -F "commit=${newrev}" \
          -F "branch=${branch}" \
          -F "author[name]=${author}" \
          -F "author[email]=${email}" \
          -F "message=${message}"
  echo ""
done

Replace username, projectname and apikey with your own details. Make the file executable, and then push a change, and a build should start!

I, for one, welcome our new screencasting overloads

Or, how I used robots to make my screencasts.

I don’t like screencasts. I don’t like watching them, and I hate making them.

However, I know a lot of people do like watching them, especially people that are trying evaluate software. It had been pointed out to me that there was no way to see the internals of 88 Miles without signing up (Personally, I would suggest signing up, but whatever). So I thought I’d investigate a way to make a screencast without wanting to stab myself in the face.

So why do I hate making them?

  1. If you change any part of the UI, you have to go through and re-record the whole thing. This becomes tedious. That said, having an out of date screencast is probably worse than having no screencast.
  2. Making typos during the recording looks unprofessional, so either you need to be perfect (not going to happen) or you need to spend ages editing out the typos (I’ve got better things to do).
  3. Not only do you have to write content, you have to record voice over audio and video. They are a lot of work.

I’ve spent a lot of time on the look and feel of 88 Miles, and I wasn’t going to produce a crappy screencast – it had to look professional. So I needed a way to automate as much of this as I could, so that I replicate the video easily.

After putting out a call to twitter, Max pointed me at a ruby gem called Castanaut that basically wraps AppleScript allowing the automation of both the screencasting software (in my case: iShowU) and Safari. Success! Sort of. There was a little bit of work to get it all working.

The first thing to do is install the gem:

gem install castanaut

I started out using the screenplay from Castanaut site, but had to make some changes to get it working. First of all, I don’t have Mousepos installed, so I removed that plugin.

Next, Castanaut seemed to miss clicks randomly, which was a pain. After a bit of digging it looks like it was the way that it was calling AppleScript. Rather than debugging that, I just installed cliclick which is a commandline app that control the mouse without AppleScript. I had to write a small plugin to override the move and click functions. (Save this to plugin/cliclick.rb)

module Castanaut
  module Plugin
    module Cliclick
      def click(btn = "left")
        `cliclick c:+0,+0`
      end

      def doubleclick(btn = "left")
        `cliclick dc:+0,+0`
      end

      def cursor(*options)
        options = combine_options(*options)
        apply_offset(options)
        @cursor_loc ||= {}
        @cursor_loc[:x] = options[:to][:left]
        @cursor_loc[:y] = options[:to][:top]

        `cliclick m:#{@cursor_loc[:x]},#{@cursor_loc[:y]}`
      end
    end
  end
end

Castanaut will use say (the built in speech synthesis software on a Mac) for timing voiceovers. You really don’t want to be using that in your final screencast, unless you are actually Stephen Hawking. To solve this problem, I wrote another plugin that automatically generates a subtitle file that, when run under VLC will display the text, allowing me to read along with the video

module Castanaut
  module Plugin
    module Subtitle
      def start_subtitles(filename)
        @filename = filename
        @start = Time.now
        @sequence = 1
        @srt = ''
        @webvtt = "WEBVTT\n\n"
      end

      def stop_subtitles
        @start = nil
        @sequence = 0
        File.write "#{@filename}.srt", @srt
        File.write "#{@filename}.vtt", @webvtt
        @srt = ''
        @webvtt = ''
      end

      def subtitle(narrative, &blk)
        start = Time.now - @start
        yield
        stop = Time.now - @start

        @srt += "#{@sequence}\n"
        @srt += "#{time_diff(start)} --> #{time_diff(stop)}\n"
        @srt += "#{narrative.scan(/\S.{0,40}\S(?=\s|$)|\S+/).join("\n")}\n"
        @srt += "\n"

        @webvtt += "#{time_diff(start).gsub(',', '.')} --> #{time_diff(stop).gsub(',', '.')}\n"
        @webvtt += "#{narrative.scan(/\S.{0,40}\S(?=\s|$)|\S+/).join("\n")}\n"
        @webvtt += "\n"

        @sequence += 1
      end

      def say_with_subtitles(narrative)
        subtitle narrative do
          say(narrative)
        end
      end

      def while_saying_with_subtitles(narrative, &blk)
        subtitle narrative do
          while_saying narrative, &blk
        end
      end

      protected
      def time_diff(time)
        micro = ((time.to_f - time.to_i) * 1000).floor
        seconds = (time.abs % 60).floor
        minute = (time.abs / 60 % 60).floor
        hour = (time.abs / 3600).floor
        (time != 0 && (time / time.abs) == -1 ? "-" : "") + hour.to_s.rjust(2, '0') + ":" + minute.to_s.rjust(2, '0') + ":" + seconds.to_s.rjust(2, '0') + ',' + micro.to_s
      end
    end
  end
end

This will create both a SRT and VTT (Web subtitle) file. Here is a screen shot of the subtitle overlayed on to the video:

Showing the subtitles overlayed in VLC

Here is an excerpt from my screenplay file:

#!/usr/bin/env castanaut
plugin "safari"
plugin "keystack"
plugin "cliclick"
plugin "subtitle"
plugin "ishowu"
plugin "sayfast"

launch "Safari", at(120, 120, 1024, 768)
url "http://88miles.net/projects"
pause 5

ishowu_start_recording
start_subtitles "/Users/myles/Movies/iShowU/tour"
pause 1
say_with_subtitles "Hi, my name is Myles Eftos, and I'm the creator of Eighty Eight Miles"
say_with_subtitles "a time tracking application for designers, developers and copywriters."
say_with_subtitles "This short video will show you how Eighty Eight Miles tracks your time"

Oh, one last thing – I found the synthesised voice was too slow, so I made another plugin that speeds up the voice (saved in plugins/sayfast.rb):

module Castanaut
  module Plugin
    module Sayfast
      def say(narrative)
        run(%Q`say -r 240 "#{escape_dq(narrative)}"`)  unless ENV['SHHH']
      end
    end
  end
end

I recorded the voice over using Audacity. I wasn’t too fussed about an exact sync, so I just hit record on Audacity, and play in VLC. If you are worried about sync, just make a noise into the microphone when hit click (tapping the mic will do it), and you can use that as a sync mark.

Protip: Don’t use the built in microphone on your laptop, unless you are going for the “I’m recording this in a toilet” aesthetic. Ideally, you’d have a decent studio mic with a pop filter (I have a Samson C01U), but you know what? A gaming headset mic will still be orders of magnitude better than your laptop microphone.

Now, you should have a MP4 and WAV file (one for video, one for audio) than need to get mashed together. I use Adobe Premier Pro for this, but iMovie works great too. You will need to remove the existing audio track from the video file as it will have the robot voice on it, replacing it with your voice over track.

Finally, I topped-and-tailed the video with some titles for that last bit of fancy.

After exporting the final render, I used FFMPEG to encode the file into a final MP4 and WEBM file so I could drop them into a video tag. To install ffmpeg:

brew install ffmpeg --with-libvpx --with-libvorbis --with-fdk-aacc

Then run the following commands

ffmpeg -i [input file] -crf 10 -b:v 1M -c:a libfaac screencast.mp4
ffmpeg -i [input file] -c:v libvpx -crf 10 -b:v 1M -c:a libvorbis screencast.webm

You can upload those files somewhere, then reference them in like so:

<video autoplay class="tour" controls height="768" preload="auto" width="1024">
<source src="/videos/screencast.mp4" type="video/mp4"></source>
<source src="/videos/screencast.webm" type="video/webm"></source>
<track default kind="captions" label="English" src="/videos/screencast.vtt" srclang="en"></track>
</video>

Want to see the output? Here is the final render embedded on the internets.

Tell your non-tech friends and family: Don’t use LinkedIn Intro

If you have friends or family that are using LinkedIn (And there are a lot of them – I’ve got family members that don’t use Facebook, that do have LinkedIn accounts), please take the time to inform them about the importance of password security.

Knowing that many of their users aren’t particularly technical, they have added a number of dubious (I’d say dangerous) techniques to bolster users connections (and by effect their userbase). Of these techniques, there are two which ask users to enter their email username and password so they can access the user’s email inbox directly.¬†This is a bad idea. Please send this post (or wholesale copy it and email it to them – I’m putting this post under Creative Commons, so copy away) to your less technical friends and family, and offer to help them fix up the mess if they have already given up their username and password.

Giving third party applications like LinkedIn your password is a bad idea.

Dear friends and family,

When you signed up for LinkedIn, they may have asked you for your email login and password, allowing them to search your contacts to create connections. By giving LinkedIn your email login and password, you have given them complete access to your email. This means they can read ALL of your email, and theoretically send email on your behalf.

You might have seen a screen like the one below:

Screen Shot 2013-10-24 at 10.55.47 AM

This is one of the services you should avoid. Even though it says that they don’t store the password, or send email on your behalf, you shouldn’t trust external services with your login and password. Ever.

LinkedIn has also announced LinkedIn Intro. This service DOES store your username and password – and it has to it in such a way that it can easily be read. This service PRETENDS to be your email server, so your email program downloads your mail from LinkedIn rather than your real server. It does this by PRETENDING to be you and logging in to your email server, downloads and changes your email to display their header. This is why they will need your username and password.

So, if you get an invite to use LinkedIn Intro, please ignore it.

What can you do if you have already signed up for these services? The easiest thing to do is to changer your email password. If you don’t know how to do it, get in contact with a trusted friend or family member that can help you out.

Please remember:¬†Anyone that has your email username and password, can read and send email on your behalf, so don’t give it to anyone that asks for it.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.

What if we treated marketing like we did code?

As someone that started writing code at a young age I, like many others, have learnt my trade via books and google searches and long hours in front of a keyboard. As the Internet engulfed our lives, solutions to problems and access to really smart people has become just one stack overflow or Github repo away. The software world really is one of knowledge sharing. It’s pretty ace.

I’ve been doing a lot of research in to marketing and sales lately for my startup, and I’ve found that the same really can’t be said for the marketing world. While there seems to be a lot of information out there, when you dig a little deeper it seems to be a rehash of a couple of ideas, a lot of link-bait lists and offers to increase my conversions by up to 250%! If I sign up to a newsletter and pay $35 a month and follow 12-simple steps that point me at a $3000 seminar.

It’s got me wondering though – would it be possible to treat marketing the same way that we treat code?

If you think about it, there are a lot of similarities between writing code, and running a marketing campaign:

  1. It’s a creative exercise. As I keep telling non-programmers, it’s not paint-by-numbers. Sure, libraries can help solve problems, but more often than not you have to engineer your own solution, or modify something else to get it working right. From what I’ve seen, marketing is the same. There is some starting points, but you need to work out what will work in a certain situation and adapt.
  2. Regardless of how well you plan, you’ll get thrown a curve-ball that means you’ll have to re-think your strategy
  3. It’s testable. Not in a unit test sense, but in a benchmark sort of way. You can do something measure it and wok out what works best in a given situation. Big-O notation for marketing, anyone? Bueller?
  4. There is a lot of self-proclaimed experts – the difference here is the output of coders can be read and assessed by anyone. Marketeers just say they are experts.

Of course, they aren’t exactly the same either:

  1. Lot’s of people make software for fun. Just look at the number of open source repos on Github. I don’t know of people that do marketing just for fun – they might find it fun, but at the end of the day, they are doing it to make money.
  2. There isn’t much actual sharing. People don’t like giving away real numbers, because they are doing this to make money and that’s a trade secret or something. It’s the equivalent of closed source software, I guess – not that there is anything wrong with it, but if it’s all closed up, it makes getting to the knowledge harder.

The question that I’ve been asking myself, is could be open source some of this stuff? Can we write up some marketing experiments and techniques, with actual results and share them for others to take inspiration off?

What is we wrote that our marketing experiments up and posted it to Github, so others could fork, implement, and improve? A library of marketing libraries for want of a better term?

Is it possible to modularise and share marketing ideas while cutting through the usual online-marketing expert bullshit? Can marketing be something that we play with for no better reason than to learn something or does it have to always just be about making a buck? What, in my n00bness have I missed, that makes this ultimately a stupid idea? Or is this actually something that could happen?

Lots of questions, not too many answers. Leave a comment, or let’s discuss it on twitter.

Ensuring background AJAX requests have completed when leaving a page

To make things feel quicker on 88 Miles, I update the UI straight after an interaction (like punching in or out), even though in reality the AJAX request is still sitting there waiting for confirmation from the server. Generally, the AJAX request will complete correctly (and if it doesn’t a message gets popped up and the UI is restored to a pre-action state), however there is a chance that the user will close their browser or navigate away before everything completes, which may result in a project staying punched in which is less than ideal.

This little snippet of JS will allow you to check for pending AJAX requests, and throw up a warning dialog is a request is still pending.

jQuery example:

var ajaxInProgress = false;
function ajaxStart() {
  ajaxInProgress = true;
}
function ajaxStop() {
  ajaxInProgress = false;
}
$(window).bind('beforeunload', function(e) {
  if(ajaxInProgress) {
    return "There is a background process that hasn't completed yet. Reloading might result in data loss.";
  }
});

Pure XHR example:

var requests = [];
var xhr = new XMLHttpRequest();
xhr.onreadystatechange = function() {
  if(this.readyState == 1) {
    requests.push(this);
  }
  if(this.readyState == 4) {
    var index = requests.indexOf(this);
    // Remove the completed request from the request list - handles out of order responses
    if(index != -1) {
      requests.splice(index, 1);
    }
  }
}
window.onbeforeunload = function() {
  if(requests.length != 0) {
    return "There is a background process that hasn't completed yet. Reloading might result in data loss.";
  }
}

Caveats

Tested on latest Chrome, Firefox, Safari and IE 9+.

It doesn’t work on the iPhone. Which is shit, because phones are the ones that need this the most. It works fine on Android and Windows Phone though.

A quick and dirty way to ensure your cron tasks only run on one AWS server

88 Miles has a cron job that runs every hour that does a number of things, one of which is billing. I currently run two servers (Yay failover!), which makes ensuring the cron job only runs once problematic. The obvious solution is to only run cron on one server, but this is also problematic, as there is no guarantee that the server running cron hasn’t died and another spun up in it’s place. The proper sysops solution is to run heartbeat or some other high-availability software to monitor cron – if the currently running server dies, the secondary will take over.

While that is the correct solution that I may implement later, it’s also a pain to setup, and I needed something quick that I could implement now. So came up with a this lo-fi solution.

Each AWS instance has a instance_id with is unique (it’s a string that looks something like this: i-235d734c). By fetching all of the instance_ids of all the productions servers in my cluster, and then sorting them alphabetically, I’ll pick the first one, and run the cron job on that. This setup uses the AWS ruby library, and assumes that your cron job fires off a rake task, which is where this code lives.

So step one is go get the ip address of instance that the job is running on.

uri = URI.parse('http://169.254.169.254/latest/meta-data/local-ipv4')
ip_address = Net::HTTP.get_response(uri).body

That URL is some magic URL that when called from any AWS instance will return metadata about the server – in this case the local ip address (also unique).

Next, pull out all of the servers in the cluster.

servers = []
ec2 = AWS::EC2.new :access_key_id => '[YOUR ACCESS KEY]', :secret_access_key => '[YOUR SECRET KEY]'
ec2.instances.each do |instance|
    servers << instance if instance.tags['environment'] == 'production'
end
servers.compact!

I tag my servers by environment, so I want to filter them based on that tag. You might decided to tag them differently, or just assume all of your server can run cron jobs – that bit is left as an exercise for the reader. Replace [YOUR ACCESS KEY] and [YOUR SECRET KEY] with your access key and your secret key.

Now, sort the servers by instance id

servers.sort{ |x, y| x.instance_id <=> y.instance_id }

Finally, check to see if the first server’s local ip address matches the ip address of the server we are currently running on

if servers.first.private_ip_address == ip_address
    # Success! We are the first server - run that billing code!
end

See? Quick and dirty. There are a few things to keep in mind…

  • If a server with a lower instance_id gets booted just as the cron jobs are due to run, you might get a race condition. You would have to be pretty unlucky, but it’s a possibility.
  • There is no testing to see if the nominated server can actually perform the job – not a big deal for me, as the user will just get billed the next day. But you can probably work around this via business logic if it is critical for your system – crons fail, so you need a way to recover anyway…

As I said – quick and dirty, but effective!

Delivering fonts from Cloudfront to Firefox

I use both non-standard and custom icon fonts on 88 Miles, which need to be delivered to the browser in some way. Since all of the other assets are delivered via Amazon’s Content Delivery Network (CDN) Cloudfront, it would make sense to do the same for the fonts.

Except that didn’t work for Firefox and some version of Internet Explorer. Turns out they both consider fonts as an attack vector, and will refuse to consume them if they are being delivered via another domain (commonly know as a cross domain policy). On a regular server, this is quite easy to fix: You just set the:

Access-Control-Allow-Origin: *

HTTP header. The problem is, S3 doesn’t seem to support it (It supposedly supports CORS, but I couldn’t get it working properly). It turns out though, that you can selectively point Cloudfront at any server, so the simplest solution is to tell it to pull the fonts from a server I control that can supply the Access-Control-Allow-Origin header.

First, set up your server to supply the correct header. On Apache, it might look something like this (My fonts are in the /assets folder):

<Location /assets>
  Header set Access-Control-Allow-Origin "*"
</Location>

If you don’t use Apache, google it. Restart your server.

Next, log in to the AWS console and click on Cloudfront, and then click on the i icon next to the distribution that is serving up your assets.

Amazon Cloudfront settings

Next, click on Origins, and then Create Origin button.

In the Origin Domain Name field enter the base URL of your server. In the Origin ID field, enter a name that you’ll be able to recognise – you’ll need that for the next step.

Hit Create, and you should see the new origin.

New origin added

Now, click on the Behaviours tab, and click Create Behavior. (You’ll need to do this once for each font type that you have)

Enter the path to you fonts in Path pattern. You can use wildcards for the filename. So for woff files it might look something like:

Setting up the cache

Select the origin from the origin drop down. Hit Create and you are done!

To test it out, use curl:

> curl -I http://yourcloudfrontid.cloudfront.new/assets/fonts.woff

HTTP/1.1 200 OK
Content-Type: application/vnd.ms-fontobject
Content-Length: 10228
Connection: keep-alive
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Date: Mon, 16 Sep 2013 06:05:35 GMT
ETag: "30fb1-27f4-4e66f39d4165f"
Last-Modified: Sun, 15 Sep 2013 17:14:52 GMT
Server: Apache
Vary: Accept-Encoding
Via: 1.0 ed617e3cc5a406d1ebbf983d8433c4f6.cloudfront.net (CloudFront)
X-Cache: Miss from cloudfront
X-Amz-Cf-Id: ZFTMU-m781XXQ2uOw_9ukQGbBXuGKbjlVsilwW44IS2MvHbeBsLnXw==

The header is there. Success! Now, pop open your site in Firefox, and you should see all of your fonts being served up correctly.

An open letter to the Liberal member for Perth: Darryl Moore.

Hi Darryl,

As a Liberal voter, today’s announcement regarding the cut to ICT spending and the implementation of a mandatory internet filter; as well as the clearly flawed coalition broadband strategy, you have forced me to swing my vote. I write you this email so you can be aware of the issues for at least one of your constituents.

It disappoints me that your party has a lack of understanding of technology and the role that it plays in the future of this country. As an IT professional, who runs my own business as well as an internet startup that exports services to the world, the short-sightedness of your strategy is concerning. We are looking backwards to support unsustainable industries like the car industry, while ignoring an industry that Australian can become a world-class player.

We should be encouraging spending in an industry that has the potential to increase exports to the world, not decreasing them.

We should also be encouraging smart people in this country to stay here, rather than forcing them to leave our country to seek opportunities – and better yet, our country should be a destination for other smart people. We should be boosting our knowledge economy, not restricting it.

I’m happy to hear you reasoning behind these decisions.

Thank you for your time.

Ref:

http://www.zdnet.com/au/australian-opposition-vows-to-implement-internet-filter-by-default-7000020270/
http://www.zdnet.com/au/coalition-to-cut-it-research-centre-nictas-funding-7000020267/

Using Selenium to generate screenshots for support documents

I’ve just been working on some support documentation for 88 Miles, and I wanted to include some screenshots. Since I’m lazy, and hate having to repeat tasks, I decided to use Selenium and Capybara to generate all the screenshots for me. Using robots means I can re-generate all my screenshots quickly, so I’m more likely to do it if I change the UI. I covered using PhantomJS for generating screenshots before, but I’m using Selenium for this to make sure a real browser is used, just in case there are any rendering differences (which there usually is).

Again, this guide is for OSX. I’m also using Chrome, rather than the default Firefox as the webdriver.

First, install the selenium server

brew install selenium-server-standalone

and start the server up

java -jar /usr/local/opt/selenium-server-standalone/selenium-server-standalone-2.35.0.jar -p 4444

Next, download the chrome webdriver from google code, unzip it and copy the chromedriver executable to /usr/local/bin

Add the selenium-webdriver gem to your Gemfile in the test section

group :test do
    gem "selenium-webdriver"
end

Edit your test/test_helper.rb and create a new driver for chrome (put this after you require capybara)

require 'capybara/rails'

Capybara.register_driver :chrome do |app|
  Capybara::Selenium::Driver.new(app, :browser => :chrome)
end

Finally, create a new test that will generate the screenshots. I created a directory for the tests, and a directory for the screen shots to be saved to

mkdir test/support
mkdir app/assets/images/support

Now you can create a integration test file, and watch as a browser opens an starts replaying all the actions!

require File.expand_path(File.join(File.dirname(__FILE__), '..', 'test_helper'))

module Support
  class ReportsControllertest < ActionDispatch::IntegrationTest
    self.use_transactional_fixtures = false

    setup do
      Capybara.default_driver = :chrome
      Capybara.javascript_driver = :chrome
    end

    teardown do
      Capybara.use_default_driver
    end

    test 'generate screenshots' do
      visit '/login'
      save_screenshot(Rails.root.join('app', 'assets', 'images', 'support', 'login.png'))
    end
  end
end

To run just this test (it’s slow, so you probably don’t want to automatically run it using guard):

bundle exec ruby -I"lib:test" test/support/reports_controller_test.rb

You can see (a beta) example on the 88 Miles beta website.

Track email opens using a pixel tracker from Rails

If you are sending out emails to you customers from your web app, it pretty handy to know if they are opening them. If you have ever sent out an email newsletter from a service like Campaign Monitor, you would have seen email open graphs. Of course, tracking this stuff is super important for a newsletter campaign, but it would also be interesting to see if users are opening, for example, welcome emails or onboarding emails.

The simplest way to do this is via a tracking pixel – a small, invisible image that is loaded off your server every time the email is opened (See caveat below). This is fairly simple to achieve using Rails by building a simple Rack application.

Caveat

This only works for HTML emails (unless you can work out how to embed an image in a plain text email), and relies on the user having “Load images” turned on. Clearly, this isn’t super accurate, but it should give you a decent estimate.

The Setup

We’ll add two models: One to tracking sending email, and one to track opening of email:

rails generate model sent_email
rails generate model sent_email_open

The schema for these are fairly simple: Save a name to identify the email, the email address it was sent to, an ip address and when the email was sent and opened.

class CreateSentEmails < ActiveRecord::Migration
  def change
    create_table :sent_emails do |t|
      t.string :name
      t.string :email
      t.datetime :sent
      t.timestamps
    end
  end
end
class CreateSentEmailOpens < ActiveRecord::Migration
  def change
    create_table :sent_email_opens do |t|
      t.string :name
      t.string :email
      t.string :ip_address
      t.string :opened
      t.timestamps
    end
  end
end
class SentEmail < ActiveRecord::Base
  attr_accessible :name, :email, :sent
end
class SentEmailOpen < ActiveRecord::Base
  attr_accessible :name, :email, :ip_address, :opened
end

With the models setup, let’s add a mail helper that will generate the image pixel – create this is /app/helpers/mailer_helper.rb

module MailerHelper
  def track(name, email)
    SentEmail.create!(:name => name, :email => email, :sent => DateTime.now)
    url = "#{root_path(:only_path => false)}email/track/#{Base64.encode64("name=#{name}&email=#{email}")}.png"
    raw("<img src=\"#{url}\" alt="" width=\"1\" height=\"1\">")
  end
end

What this does is give our mailers a method called track that takes a name for the email and the email address of the person we are sending it to. To enable it, call the helper method in the mailers you want to track:

class UserMailer < ActionMailer::Base
  helper :mailer
end

Now we can add the tracker to our html emails. Say we have a registration email that goes out, and there is a user variable, with an email attribute:

<!-- Snip -->
<%= track('register', @user.email) %>
<!-- Snip -->

Right, now the magic bit:

Create a directory called /lib/email_tracker and create a new file called rack.rb

module EmailTracker
  class Rack
    def initialize(app)
      @app = app
    end

    def call(env)
      req = ::Rack::Request.new(env)

      if req.path_info =~ /^\/email\/track\/(.+).png/
        details = Base64.decode64(Regexp.last_match[1])
        name = nil
        email = nil

        details.split('&').each do |kv|
          (key, value) = kv.split('=')
          case(key)
          when('name')
            name = value
          when('email')
            email = value
          end
        end

        if name && email
          SentEmailOpen.create!({
            :name => name,
            :email => email,
            :ip_address => req.ip,
            :opened => DateTime.now
          })
        end

        [ 200, { 'Content-Type' => 'image/png' }, [ File.read(File.join(File.dirname(__FILE__), 'track.png')) ] ]
      else
        @app.call(env)
      end
    end
  end
end

Create a 1×1 pixel transparent PNG and save it as track.png and place it in the same directory. Next, include it in your config/application.rb

module App
    class Application < Rails::Application
        require Rails.root.join('lib', 'email_tracker', 'rack')
        # Some other stuff
        config.middleware.use EmailTracker::Rack
    end
end

And that’s it! Now, every time the email gets sent out, it will create a record in the send_emails table, and if it is opened (and images are turned on) it will create a record in send_email_opens. Doing up a status board is left up as an exercise to the user, but you can check you percentage open rate by doing something like:

(SentEmailOpen.where(:name => 'register').count.to_f / SentEmail.where(:name => 'register').to_f) * 100

How it works

It’s super simple. The track method generates a Base64 encoded string that stores the name of the email and the email address it is being sent to. It them returns an image URL that can be embedded in the email. The Rack app looks for a URL that looks like /track/email/[encodedstring].png and if it matches records the hit. It then returns a transparent PNG to keep the email client happy.

I might get around to turning this into a gem if there is enough interest.

Next