@madpilot rants

Track email opens using a pixel tracker from Rails

If you are sending out emails to you customers from your web app, it pretty handy to know if they are opening them. If you have ever sent out an email newsletter from a service like Campaign Monitor, you would have seen email open graphs. Of course, tracking this stuff is super important for a newsletter campaign, but it would also be interesting to see if users are opening, for example, welcome emails or onboarding emails.

The simplest way to do this is via a tracking pixel – a small, invisible image that is loaded off your server every time the email is opened (See caveat below). This is fairly simple to achieve using Rails by building a simple Rack application.

Caveat

This only works for HTML emails (unless you can work out how to embed an image in a plain text email), and relies on the user having “Load images” turned on. Clearly, this isn’t super accurate, but it should give you a decent estimate.

The Setup

We’ll add two models: One to tracking sending email, and one to track opening of email:

rails generate model sent_email
rails generate model sent_email_open

The schema for these are fairly simple: Save a name to identify the email, the email address it was sent to, an ip address and when the email was sent and opened.

class CreateSentEmails < ActiveRecord::Migration
  def change
    create_table :sent_emails do |t|
      t.string :name
      t.string :email
      t.datetime :sent
      t.timestamps
    end
  end
end
class CreateSentEmailOpens < ActiveRecord::Migration
  def change
    create_table :sent_email_opens do |t|
      t.string :name
      t.string :email
      t.string :ip_address
      t.string :opened
      t.timestamps
    end
  end
end
class SentEmail < ActiveRecord::Base
  attr_accessible :name, :email, :sent
end
class SentEmailOpen < ActiveRecord::Base
  attr_accessible :name, :email, :ip_address, :opened
end

With the models setup, let’s add a mail helper that will generate the image pixel – create this is /app/helpers/mailer_helper.rb

module MailerHelper
  def track(name, email)
    SentEmail.create!(:name => name, :email => email, :sent => DateTime.now)
    url = "#{root_path(:only_path => false)}email/track/#{Base64.encode64("name=#{name}&email=#{email}")}.png"
    raw("<img src=\"#{url}\" alt="" width=\"1\" height=\"1\">")
  end
end

What this does is give our mailers a method called track that takes a name for the email and the email address of the person we are sending it to. To enable it, call the helper method in the mailers you want to track:

class UserMailer < ActionMailer::Base
  helper :mailer
end

Now we can add the tracker to our html emails. Say we have a registration email that goes out, and there is a user variable, with an email attribute:

<!-- Snip -->
<%= track('register', @user.email) %>
<!-- Snip -->

Right, now the magic bit:

Create a directory called /lib/email_tracker and create a new file called rack.rb

module EmailTracker
  class Rack
    def initialize(app)
      @app = app
    end

    def call(env)
      req = ::Rack::Request.new(env)

      if req.path_info =~ /^\/email\/track\/(.+).png/
        details = Base64.decode64(Regexp.last_match[1])
        name = nil
        email = nil

        details.split('&').each do |kv|
          (key, value) = kv.split('=')
          case(key)
          when('name')
            name = value
          when('email')
            email = value
          end
        end

        if name && email
          SentEmailOpen.create!({
            :name => name,
            :email => email,
            :ip_address => req.ip,
            :opened => DateTime.now
          })
        end

        [ 200, { 'Content-Type' => 'image/png' }, [ File.read(File.join(File.dirname(__FILE__), 'track.png')) ] ]
      else
        @app.call(env)
      end
    end
  end
end

Create a 1×1 pixel transparent PNG and save it as track.png and place it in the same directory. Next, include it in your config/application.rb

module App
    class Application < Rails::Application
        require Rails.root.join('lib', 'email_tracker', 'rack')
        # Some other stuff
        config.middleware.use EmailTracker::Rack
    end
end

And that’s it! Now, every time the email gets sent out, it will create a record in the send_emails table, and if it is opened (and images are turned on) it will create a record in send_email_opens. Doing up a status board is left up as an exercise to the user, but you can check you percentage open rate by doing something like:

(SentEmailOpen.where(:name => 'register').count.to_f / SentEmail.where(:name => 'register').to_f) * 100

How it works

It’s super simple. The track method generates a Base64 encoded string that stores the name of the email and the email address it is being sent to. It them returns an image URL that can be embedded in the email. The Rack app looks for a URL that looks like /track/email/[encodedstring].png and if it matches records the hit. It then returns a transparent PNG to keep the email client happy.

I might get around to turning this into a gem if there is enough interest.

Freelancer mentoring

I’ve been a freelancer for 85% of my work career. I’ve traversed the giddy-heights and the earth-shaking lows of working for myself for a long time, and I think it’s time to give back. One thing that I have noticed during those 11 years is that it is all to easy to live in your little bubble and not talk to anyone other than clients for weeks on end. So, I’m going to start running freelancer mentoring.

I want to pull together a small group of freelancers, and then have a meet-up (probably 4-5 people to start with) on a Friday morning at my office or the coffee shop next door, and basically just chat about how things are going, give some advice, but generally act as a sounding board. Kind of like a water cooler for people that don’t own a water cooler (do people still own water coolers?). Having a small group means you can form a support group for each other too; as well as perhaps team up on bigger jobs.

Oh, and this isn’t just for web people – the basic tenants of freelancing is the same regardless of industry, and you’ll be amazed how helpful talking shop to freelancers from other industries can be.

To start with we’ll probably meet once a month.

Anyway, if you are a freelancer in Perth, Western Australia and are craving some human interaction, either hit me up in the comments, via twitter or email me directly.

Testing UI with Integration tests in Rails with phantomjs

Computers aren’t very good at testing visual changes on websites. Sure you can test the differences between two images, but image changes are really brittle. However, humans are pretty good at it. As a result, the common way to test visuals it to actually click through stuff on a browser. This doesn’t scale. For one, if your UI changes based on different initial states, it can take ages to set this stuff up.

Thankfully there is a fairly simple solution.

Rails integration tests can control with a number of browsers using Capybara and a automation system like Selenium or PhantomJS. If you haven’t heard of these, Selenium automates real browsers, and PhantomJS is a headless webkit browser. Both support modern CSS and JavaScript, so it’s like having a 1000 monkeys on a 1000 UNIX terminals. It just so happens that both also support taking screenshots, which we can use as a poor-man’s Mechanical Turk.

Aside: There is another solution: Using something like BrowserStack, although because it works remotely, it might make setting up the testing environment between tests difficult. I’ll look at this later in another blog post once I get a chance to play with it.

There are pros and cons to both systems:

Selenium

Pro: Can control basically any browser that it has a driver for – including Android and iPhones.
Con: It requires Java and a computer with all of those browsers and emulators installed. It’s also the slower of the two.

PhantomJS

Pro: Is headless, so doesn’t require X windows, or any other window manager.
Con: It’s WebKit only.

I’ll go through setting up PhantomJS on OSX, because that’s what I’ve set up so far; I’ll leave setting up Selenium as an exercise for the reader – the theory should be the same.

First, you need to install PhantomJS. On OSX, it’s just a matter of using homebrew

brew install phantomjs

Next, add the poltergeist gem (which is an interface to PhantomJS. It will also install Capybara)

group :test do
    gem "poltergeist", "~> 1.3.0"
end

Add the following to your test/test_helper.js

require 'capybara/rails'
require 'capybara/poltergeist'

Capybara.default_driver = :poltergeist
Capybara.javascript_driver = :poltergeist

class ActionDispatch::IntegrationTest
  include Capybara::DSL

  setup do
    Capybara.reset!
  end

  teardown do
    Capybara.reset!
  end
end

Finally, generate a integration test

rails generate integration_test name_of_the_test

Open up the Integration test and create a test

require 'test_helper'

class NameOfTheTestIntegrationTest < ActionDispatch::IntegrationTest
    # Don't forget this line - Database transactions don't work well with Capybara
    self.use_transactional_fixtures = false

    test 'some test name' do
      visit '/'
      # This is the important line. This is what saves the screenshot
      save_screenshot(Rails.root.join('tmp', 'screenshots', 'name_of_the_integration', 'some-test-name.png'))
    end
end

Run the integration test, and you will find a nice 1024×768 PNG screen shot in the tmp/screenshots/name_of_the_integration/some-test-name.png directory.

So, after your tests run, you can scan through the screen shots and see if anything looks weird. Win!

For a more in-depth instructions for running tests in Capybara, checkout their GitHub page.

So… It’s been a while.

Yeah, so I’ve decided to start blogging again.

While twitter is good for short bursts of vitriol, I’m missing the longer form of the game, and I think it is time to get back to it.

So what have I been up to for the past year-and-a-half? Besides my consulting and freelancing work, Adrianne and I started (and have almost wrapped up) Horse & Cart; Me and some other guys built a landing page for a product over a weekend (and got a shit-ton of sign ups); I started working on 88 Miles again, with the intention of doing 2 hours a day on it, which I was doing really well until I bought and subsequently had to move into a house. That stuff is a serious time sink. Who knew. (It’s ok, though – I’m back on 88 Miles again. New version due on 15 September!)

Any way, I managed to find a couple of hours to throw some paint around here, upgrade WordPress (which I’m glad to see still looks like it was coded by moneys throwing faeces at a keyboard – I don’t know why I do this to myself) and get this site uploaded to a new server. Oh – don’t bother reporting problems with this site on IE < 9 – I don’t care.

My intention is to continue blogging about tech, code and business stuff, just like before. And of course, you can still find my pithy rants on my twitter account.

Speak soon,

Myles.

Simulating the iOS “slider” checkbox, using just CSS

… well, some CSS and one image.

I’ve been really interested in emulating the iOS UI in HTML, CSS and JS. Not for any useful reason, more out of morbid curiosity. It could come in handy for home screen web applications, but for standard webapps, I’m beginning to think the uncanny valley plays too big a role to give good results. Regardless, it’s something I’m still playing with.

One of the elements that is quite different between Mobile Safari and native iOS applications is the rendering of checkboxes. Mobile Safari renders them in a similar way to desktop Safari – as a traditional checkbox, but native iOS application render a slider:

Example of the slider in iOS

There have been a number of techniques to simulate this in the browser: the most popular is to replace the checkbox with a couple of spans. The problem with this technique is that, unless you code them in via JavaScript, you lose all of the default events that occur, and even if you do, you are bound to forget one.

By accident, I found that you can clear the default styling of a checkbox using the -webkit-appearance: none CSS rule. Which gives us a completely blank state to work with. So let’s start with what the checkbox looks like without any styling:

The default style for checkboxes in iOS

Next, let’s add  -webkit-appearance: none – you’ll notice the checkbox disappears

iOS checkbox with -webkit-appearance: nonw

Next, we’ll add a background graphic that looks like the slider. This is the one I did. It is terrible, but it’ll do for this example. You’ll notice that it has both the “On” and “Off” sides in one graphic. More on that in a second.

Checkbox sprite

So now we can drop in the CSS to style it:

input[type=checkbox] {
  border: 1px solid #686868;
  height: 27px;
  width: 79px;
  -webkit-appearance: none;
  -webkit-border-radius: 13px;
  background-image: url(/blog/wp-content/uploads/2011/12/checkbox-background.png);
  background-position: -52px 0;
  background-repeat: no-repeat;
  -webkit-animation-timing-function: ease-in-out;
  -webkit-animation-duration: 400ms;
  -webkit-animation-name: checkbox-switch-off;
}

input:checked[type=checkbox] {
  background-position: 0 0;
  -webkit-animation-name: checkbox-switch-on;
}

@-webkit-keyframes checkbox-switch-on {
  from {
    background-position: -52px 0;
  }
  to {
    background-position: 0 0;
  }
}

@-webkit-keyframes checkbox-switch-off {
  from {
    background-position: 0 0;
  }
  to {
    background-position: -52px 0;
  }
}

So what is happening here? It’s pretty straight forward actually. We put a border, and set the height and width appropriately, then set a border radius (we are simulating iOS 5 here, so they are rounded). Next we add our background images, and set the background position so that the “Off” state is showing, using regular old CSS spriting.

Next, we target the checked state, using the :checked pseudo class. This will move the background image to the “On” state when it is checked.

Finally, we add some a slide animation. We create a webkit animation that tweens the background image from one position to the other. Simple really!

Demo

The final style

Problems

  1. It’s  webkit only at the moment. If you can zero out checkbox styling on Firefox or Opera, it wouldn’t be difficult to implement for them too (Just change the vendor specific CSS and animations). It might also work in IE 10?
  2. The animation will always go from On to Off on page load, because the CSS animation will run. You could easily fix this with a line of JavaScript, but if anyone else can suggest a way to do it purely in CSS, I’d be all ears.

How to re-play an AJAX request in jQuery after an authentication error

I’m building a mobile web app that is basically one HTML file + backbone.js + other JavaScript magic. The app is authenticated, so the user needs to be able to login. Thankfully the server returns a 403 if a request is not authorised to access the requested endpoint. Ideally, if that happened, a login form would pop up, the user would enter their details, and the system would continue on it’s merry way.

With jQuery, this is surprisingly easy.

You can setup global settings using the jQuery.ajaxSetup method. These settings get injected into every AJAX request you make via the jQuery API. The attribute we are interested in setting is the statusCode attribute: in particular the 403 function.

jQuery.ajaxSetup(
  statusCode: {
    403: function() {
      // This calls a backbone view that renders a login window, then
      // calls the success function one the user has been authenticated
      // The call back simply re-runs $.ajax using the current context object,
      // which conveniently is a hash of the original AJAX request params.

      var sessionLogin = new SessionsLoginView();
      var context = this;
      sessionLogin.render({
        success: function() {
          $.ajax(context);
        }
      });
    }
  }
});

What happens is a 403 response code gets captured, we save the current AJAX object into the context variable, we then call the login form, which gets the user to authenticate and then calls the success function. It just so happens that the context variable stores a hash of the original AJAX request settings, which we pass directly into $.ajax. This in effect replays the query, now with the new authentication token!

The beauty of this (besides the fact that it’s hands off, and nicely abstracted) is that it will handle changing passwords on the server gracefully (it’ll just pop up the login form again), as well as handle incorrect usernames or passwords.

Time to learn Ruby on Rails!

I like Rails. A lot. And you should too – it makes build web stuff fun, and faster. It’s poetry when compared to PHP. Not to mention there is some smart nerds doing work on it, and it has one of the most vibrant and passionate communities around. Unfortunately, the learning curve can be significant, especially if you are learning Ruby at the same time. Me to the rescue!

The guys behind Sitepoint have just launched a new site: Learnable, which is an online training centre, where people not only go to learn stuff, but they can teach stuff too! They ask me to create a Ruby on Rails 3 course for their launch, and it just went live today!

The course is 12 lessons, and covers enough stuff to build a real app – a code snippet library (A bit like pastie or github gists). I’ll be maning the course forums and answering any questions you have, and will appear online via video link up for some live Q&A sessions in the next couple of weeks.

If you are still using PHP, or have been meaning to learn Rails for a while, then go and sign up – it’s only $19.95 (Bargin!).

The only real pre-requisite is a basic grasp of programming – if you understand if statements, for loops and variables, you should be fine. Serious. Go now. It’s awesome. But don’t just believe me, listen to this talking headshot video of me:

https://learnable.com/courses/learning-rails-3-212

Need a frontend coder?

While I work on some personal summer projects, I’m looking at getting some front end contract work – you know HTML/CSS/JS sort of stuff.If you know anyone looking to turn pictures into websites, get them in contact with me.Hourly rate negotiable depending on contract length.

Making the OSX Terminal.app work properly

I just acquired a 13″ MacBook so I can build some me iPhone and iPad apps, and the allure of Unix on the desktop is a nice added benefit, however it is apparent that the default terminal emulator is kind of balls out of the box. I spend A LOT of time in a terminal, with VIM being my IDE is choice. What many people might not know is that modern terminal emulators support mouse gestures, as does vim (:set mouse=a).

I use this. All the time.Vim 7 supports files tabs, which allows you to open multiple files at once (command :tabe <filename>). Vim being vim, you can navigate your tabs using key sequences (Next Tab :tabn Previous Tab :tabp Move to tab :tabm <number> et al.), but clicking stuff with your mouse is easier.The scroll wheel is also quite useful, which vim also supports.Finally, click and dragging over text in vim with a mouse enters Visual mode, which allows you to yank and delete blocks of text, also quite handy.The problem is, Terminal.app doesn’t support mouse stuff… Out of the box any way.

Step 1: Download and install SIMBL

Step 2: Download and install MouseTerm

Step 3: Restart Terminal.app

Right, so that is sorted, next up: OSX’s stupid keybindings. Ok, I guess it makes sense for the home key to go to the very beginning of the document and end to do the opposite, but really the Windows way of home and end placing the cursor to the beginning and end of the line is more practical. Also, PgUp and PgDn don’t work. So, open Terminal.app then select Preferences from the Terminal menu. Click the Keyboard tab, and paste the following strings into the areas next to the Keys:

home: 33[H
end: 33[F
pgup: 33[5~
pgdn:  33[6~

FINALLY, these binding don’t place nice with vim, so open up .vimrc and enter the following:

set mouse=a
map ^[[F $
imap ^[[F ^O$
map ^[[H g0
imap ^[[H ^Og0

Creating a bootable USB stick to install VMware ESXi

I had to install VMware ESXi on a machine that didn’t have a CDROM or DVD player. For extra difficulty, I also didn’t have a Linux box with a USB drive handy – but after a bit of hair pulling, I managed to create a bootable USB stick image in Linux (In my case Gentoo). Hat tip to http://communities.vmware.com/thread/75792

  1. Create a fake filesystem. I created a 1Gb, but 512Mb should be enough
    dd if=/dev/zero of=/mnt/vmware.img bs=1024 count=1048576
    
  2. Associate it with a loopback device
    losetup /dev/loop0 vmware.img
    
  3. Create a FAT filesystem on the image
    mkfs.vfat /dev/loop0
    
  4. Mount the VMware CD ISO to a temporary directory
    mkdir -p /mnt/vmware
    mkdir -p /mnt/vmware-usb
    mount -t iso9660 -o loop /path/to/vmware.iso /mnt/vmware
    mount -t vfat -o loop /mnt/vmware.img /mnt/vmware-usb
    
  5. Copy the contents across
    cp -r /mnt/vmware/* /mnt/vmware-usb
    
  6. Delete the isolinux.bin and rename isolinux.cfg file on the USB flash disk to syslinux.cfg in /mnt/vmware-usb
  7. Add usb to the end of the append line in syslinux.cfg
  8. Run syslinux to write a bootloader
    syslinux -s /dev/loop0
    
  9. Unmount everything
    cd /mnt
    unmount /mnt/vmware
    unmount /mnt/vmware-usb
    losetup -d /dev/loop0
    

vmware.img should now be a bootable USB image. You will need to write it to your USB stick now. In linux you can use dd (assuming your USB stick is at sdb):

dd if=/mnt/vmware.img of=/dev/sdb

In windows, use Image Writer for Windows.Or, an easier solution might be to buy a DVD Drive :)

Previous Next