@madpilot makes

WGET: The poor man’s SVN – Using Capistrano on a host with out subversion

I have a client for whom I created a CakePHP-based website for over a year ago. He has since come back to me and asked for a number of changes. I thought I would take the opportunity to use capistrano, because there are a number of steps I always had to perform when updating his site and I hate having to do them manually.

I went about checking all the necessary requirements on his host:

  1. SSH access – check! The host his site was on allows an SSH connection which is required by capistrano
  2. Apache follows symbolic links – check! Because capistrano uses a symbolic link from the document root to the latest version of the site, Apache needs to be able to follow them (i.e the site’s apache configuration needs FollowSymLinks enabled)
  3. Has svn installed – fail! This could be a problem. Capistrano by default checks out the HEAD revision from the defined repository – if it can’t use SVN, it can’t download the latest version of the site.

So close! If only I could download the HEAD revision of a site using a common command line system. I thought about writing a SVN-to-web interface, that would check out the latest version and post them as a website, but then I remembered SVN does that out of the box using the SVN apache module. Thankfully, when I was building my development machine, I made I installed the SVN module – it was now time to use it!

First I needed to tell Apache serve up the a copy of the SVN repository. Dropping the following into the Apache config file did the trick:

  1. <location /url/you/would/like/to/access>
  2. DAV svn
  3. SVNPath /path/to/svn/repository
  4. AuthType Basic
  5. AuthName “My Secret SVN Repository”
  6. AuthUserFile /path/to/a/.htpassword/file
  7. Require valid-user
  8. </Location>

For those playing at home, replace /url/you/would/like/to/access with a nice easy path – this is the URL you will access to download the files, replace /path/to/svn/repository with the actual physical path to your repository and create a .htpassword file so you can limit access to the repository by using the htpasswd2 command: htpasswd2 -c /path/to/a/.htpassword/file username would work in this case (After substituting a username and real path, of course)

If you point you browser to the URL you just setup, you should see the root directory of the repository, after you enter the username and password you setup. Congratulations! You are basically there. Now you just need to reconfigure your capistrano to use wget instead of svn. I do this by overriding the deploy method – because I’m not using rails for this project, the paths and shard folders are different anyway. If you are using rails, you might need to have a look at the original recipe file and replace the svn command with the one below.

  1. task :deploy do
  2. run “wget –user=#{wget_user} –password=#{wget_pass} -m –cut-dirs=4 -nH -P #{release_path} -q -R index.html #{repository}”
  3. run “ln -nfs #{release_path} #{current_path}”
  4. run “rm -rf #{release_path}/app/webroot/files”
  5. end

The only modification to that line is the number after the –cut-dirs switch – it should be equal to the number of directories in the URL. In our example the URL is /url/you/would/like/to/access so –cut-dirs it needs to be equal to 6.

The last thing to do is to setup the wget_user and wget_pass variables to be equal to the username and password you created using htpasswd2.

That should do it! You can now deploy to a server that is sans SVN!

Caveats: Because of the way the SVN module and WGET work, I’ve had to not include he downloading of index.html (Basically WGET treats the directory listing as a page, and will output it as index.html) so this technique will not work if you have any pages called index.html in your structure. Work around: Rename all instances of index.html to index.htm

You might get some weird results if some one checks in code at the same time as you do a deploy – unless you have a bucket load of developers working on your system and you have no communication between developers, this is pretty unlikely.

(Names have been changed to protect the innocent)

Make scheduling jobs easier – use web services!

Web developers deal with scheduled jobs a lot. Any online application that deals with paid subscriptions needs to remind users when they need to pay up. Other apps may require data mining services to be run on a nightly basis.

This is pretty easy to do – run a cronjob under *nix or a scheduled task under windows. However, where things get tricky is when you have spent an in ordinate amount of time coding business logic into you application. Duplicating this logic for an external script to be run by cron is pretty silly, not to mention bug-prone. However there is a quick and simple solutions: Web services!

By setting up a web service that does your maintainence, you can leverage the classes and business rules that you have already written. For example, if we take a Model-View-Controller framework (such as Rails or CakePHP) we may have logic that will do data manipulations before the data is saved to the database. There is no way I would want to try and emulate what these frameworks do in an external script. By using web services, you are calling the framework natively and you avoid all of these problems.

How it works

It is really simple:

  1. Setup a webservice that will perform the functions you need to run on a regular basis.
  2. Create a simple script that calls the web services.
  3. Add this script to you crontab or task scheduler

The beauty is that you don’t even need to bother setting up the web service using SOAP or XML-RPC – REST will do the job quite nicely, especially since the web service is designed to only be used by you. However, when using rails I like to use SOAP, because it is so easy to setup and use. Here is a simple example:

  1. class HousekeepingController < ApplicationController
  2. wsdl_service_name ‘Housekeeping’
  3. web_service_api HouseKeepingAPI
  4. def maintain()
  5. # Run you maintainence script here
  6. end
  7. end

This file is the webservices controller. You would add your logic here – this may be expiring users or automatically checking email.

  1. class HousekeepinApi < ActionWebService::API::Base
  2. api_method :maintain, :returns => [:string]
  3. end

This file defines the method to the web service caller. I usually set the return to type :string so I can output statistical messages or errors that occur during the job.

  1. #!/usr/bin/ruby -w
  2. require ‘soap/wsdlDriver’
  3. include SOAP
  4. begin
  5. wsdl = ‘path_to_wsdl_file’
  6. factory = WSDLDriverFactory.new(wsdl)
  7. housekeeping = factory.create_rpc_driver
  8. out = housekeeping.maintain
  9. put_s out
  10. rescue Exception
  11. $stderr.print “An error occured: #{$!}n
  12. $stderr.print “Detailes error description, if any:n
  13. raise
  14. end

This file is called by the cronjob. You will need to replace the string path_to_wsdl_file with the path to the real WSDL file. You get get the WSDL file from a Rails web service by querying /:controller/wsdl (so in this example http://www.yoursite.com/housekeeping/wsdl – pipe the returned xml to a file and save it.

Then it is just a matter of adding the housekeeping_run.rb command to you cronjob! Cron will even email you the results that get returned by your service – Nifty!

Nice an easy, eh?

Update: Improved validation in CakePHP for version 1.x

I recently started a new project in CakePHP and thought it time to use version 1. But it seems the way that the validator code works has changed, so here is the updated code:

The following goes in app_model.php

Updated validator code
  1. function invalidFields ($data = array())
  2. {
  3. if (empty($data)) {
  4. $data = $this->data;
  5. }
  6. if (!$this->beforeValidate()) {
  7. return false;
  8. }
  9. if (!isset($this->validate)) {
  10. return true;
  11. }
  12. if (!empty($data)) {
  13. $data = $data;
  14. } elseif (isset($this->data)) {
  15. $data = $this->data;
  16. }
  17. if (isset($data[$this->name])) {
  18. $data = $data[$this->name];
  19. }
  20. $errors = array();
  21. foreach($this->validate as $field_name => $validators) {
  22. foreach($validators as $validator) {
  23. if (isset($data[$field_name]) && !preg_match($validator[‘expression’], $data[$field_name])) {
  24. $errors[$field_name] = $validator[‘message’];
  25. }
  26. }
  27. }
  28. $this->validationErrors = $errors;
  29. return $errors;
  30. }

The validation helper method (See my previous post) looks like this:

  1. function tagErrorMessages() {
  2. $messages = “”;
  3. if(isset($this->validationErrors)) {
  4. foreach($this->validationErrors as $tag) {
  5. foreach($tag as $element => $message) {
  6. $messages .= sprintf(SHORT_ERROR_MESSAGE, empty($message) ? ‘Error in field: ‘ . $element : $message);
  7. }
  8. }
  9. }
  10. return $messages;
  11. }

3rd Degree e-news site publishes it’s first edition

The Edith Cowan University journalism students have launched the first edition of 3rd Degree – an online e-news site. The site was designed by Paul Bui and was developed by me in CakePHP.

The site allows the 3rd year students to understand the pressure of publishing a weekly news publication, with different teams controlling different parts of the process. If you would like to receive the weekly newsletter, you can register here.

3rd Degree is the brainchild of Kayt Davies, who is the lecturer in the unit.

New port80 event announced.

Port80 – the Australian Web Industry, of which I’m the membership officer has announced the next event – Ideas3.

The event features John Allsop from Sydney, who is a directory of Westciv, creators of Stylemaster; and Mark Boulton from the UK who is a noted typographer and designer who is currently working at the BBC.

For those of you who aren’t in Perth for the event on April 11, we will be posting the podcasts and photos after the event.

As an aside, the new Port80 site, which drives the event and memeber management systems was written by me in CakePHP.

Interested in starting a Port80 group in your local area? We are currently preparing an information pack. Port80 started as a way for web designers and developers in Perth could catch up in a casual, informal environment – usually at a pub. It is amazing how well it works – it’s a great way to meet other like minded people. You can read a better history on the Port80 website. Don’t forget to check out the port80 forums too!

Adding automatic JavaScipt validators to CakePHP

I’m reposting the validators.php file that I uploaded in my previous post, but this time adding a function that will automatically generate a javascript function called validate() that returns an array of error strings. This is made possible because CakePHP’s validation model uses regular expressions.

NOTE: This function has been written using my modified validator class. Please read my previous post to see what it is all about.

The strings and expressions are identical to the ones you get by callling $this->modelName->validates() in a controller, so you can avoid a round trip to the server in many cases.

To use, drop the validators.php into your app helpers directory (as before), and call the function $validators->javascriptErrors(‘modelName’); where modelName is the name of the model you want to pull the validation data from. For those of you playing at home, you may notice that the $modelNames variable gets converted to an array – this is because you can actually supply an array of modelNames if you need to validate the inputs from multiple models. Why? I had a need for it in a project, which I might go through later :)

Now, this isn’t quite complete, because it only returns an array of strings, you still need to output the string somehow. I usually output the strings to div, as I hate JavaScript alerts. The code for which is here. In that file I have defined a function called validateForm(form); which updates the div and returns false on an error, so you can use it in the onSubmit handler for the form.

Improved validation using CakePHP

I quite like CakePHP – in fact, I find it rather difficult to do site without it now. It makes everything so much easier, especially the ActiveModel classes. However, I have found the validation model a little lacking. The way it is currently implemented only allows one validation expression for each database column, and you need to re-write the validation message on each page the model is used on. This is quite restrictive and error prone – if you need to add a field to the database, you have to update more than one page.

I have managed to modify the way the validators work, so that you can define a message and have multiple validation expressions for each field.

I used the modifications on the AccountancyAge Careers Fair by modifying the model.php file that lives in the Cake library directory and it worked really well, but I wanted to set it up so that I didn’t have to modify these files everytime – It is really inconvienient to have to re-make the changes when updating the library. This is expecially a problem at the moment, because there seems to be a new release candidate every other day (Which is great – can’t wait for version 1!)

Anyway, after a little digging, I found that it is possible to override the app_model.php file in the same way as you can override the default app_controller.php file. By placing a copy of the file in the app/ directory, Cake will use this file instead.

The modification

The modification as it stands overrides the invalidFields function of model.php (this is based on RC3 – RC4 should work ok as well). You can download the new app_model.php here.

To define validation rules, you add an associative array with the paramaters expression and message. This is probably best explained using an example. Say you have a required field called “post_code” that must not be empty and must be 4 digits, you could do the following:

var $validate = array(

‘post_code’ => array(

array(‘expression’ => VALID_NOT_EMPTY, ‘message’ => ‘You have not entered a post code’),

array(‘expression’ => ‘/d{4}/’, ‘message’ => ‘A postcode must be four digits’)



Easy as! There is one last thing though. To make it easier to display the messages, I have created the following user helper called validators.php (Drop it in to app/views/helpers) (Download validators.php here)

To use, you put the function < ?php echo $validators->tagErrorMessages() ?> (Don’t forget to include Validators in your helper array) which will generate a block message wrapped in div tags with the class “message”. You can customise this file if you don’t like the way it is displayed.

Happy baking :)

Hmmm, CakePHP

As many of you would know, I’m playing with Ruby on Rails at the moment. I am really liking the MVC model that is presents, however, my biggest gripe about it, is that there is very little support hosting wise. Pixelbox over here in Perth is hosting it, but that doesn’t really help me for those clients that don’t want to migrate. I also feel a little loathed to move clients sites to a new technology as the support is not there yet.

Shane over at Bam sent me a link to CakePHP which looks really interesting. It started as a port of rails for PHP, but they are starting to evolve into a different beast. I’ve been playing with it for the last couple of days and I must say I like it. It is still a little rough around the edges, but it hasn’t hit beta yet, so give it time. The documentation is also a little sparse, but it’ll catch up.

I really like the fact that it runs on both PHP4 and PHP5. I think it may be something I will use a lot more once it becomes stable.

I’m planning on doing the Port80 membership system in it.