WGET: The poor man’s SVN - Using Capistrano on a host with out subversion

Posted on March 17th, 2007 in capistrano, cakephp, software design process by Myles Eftos || 1 Comment

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

I have a client for whom I created a CakePHP-based website for over a year ago. He has since come back to me and asked for a number of changes. I thought I would take the opportunity to use capistrano, because there are a number of steps I always had to perform when updating his site and I hate having to do them manually.
I went about checking all the necessary requirements on his host:

  1. SSH access - check! The host his site was on allows an SSH connection which is required by capistrano
  2. Apache follows symbolic links - check! Because capistrano uses a symbolic link from the document root to the latest version of the site, Apache needs to be able to follow them (i.e the site’s apache configuration needs FollowSymLinks enabled)
  3. Has svn installed - fail! This could be a problem. Capistrano by default checks out the HEAD revision from the defined repository - if it can’t use SVN, it can’t download the latest version of the site.

So close! If only I could download the HEAD revision of a site using a common command line system. I thought about writing a SVN-to-web interface, that would check out the latest version and post them as a website, but then I remembered SVN does that out of the box using the SVN apache module. Thankfully, when I was building my development machine, I made I installed the SVN module - it was now time to use it!

First I needed to tell Apache serve up the a copy of the SVN repository. Dropping the following into the Apache config file did the trick:

  1. <location /url/you/would/like/to/access>
  2. DAV svn
  3. SVNPath /path/to/svn/repository
  4. AuthType Basic
  5. AuthName “My Secret SVN Repository”
  6. AuthUserFile /path/to/a/.htpassword/file
  7. Require valid-user
  8. </Location>

For those playing at home, replace /url/you/would/like/to/access with a nice easy path - this is the URL you will access to download the files, replace /path/to/svn/repository with the actual physical path to your repository and create a .htpassword file so you can limit access to the repository by using the htpasswd2 command: htpasswd2 -c /path/to/a/.htpassword/file username would work in this case (After substituting a username and real path, of course)

If you point you browser to the URL you just setup, you should see the root directory of the repository, after you enter the username and password you setup. Congratulations! You are basically there. Now you just need to reconfigure your capistrano to use wget instead of svn. I do this by overriding the deploy method - because I’m not using rails for this project, the paths and shard folders are different anyway. If you are using rails, you might need to have a look at the original recipe file and replace the svn command with the one below.

  1. task :deploy do
  2. run “wget –user=#{wget_user} –password=#{wget_pass} -m –cut-dirs=4 -nH -P #{release_path} -q -R index.html #{repository}”
  3. run “ln -nfs #{release_path} #{current_path}”
  5. run “rm -rf #{release_path}/app/webroot/files”
  6. end

The only modification to that line is the number after the –cut-dirs switch - it should be equal to the number of directories in the URL. In our example the URL is /url/you/would/like/to/access so –cut-dirs it needs to be equal to 6.

The last thing to do is to setup the wget_user and wget_pass variables to be equal to the username and password you created using htpasswd2.

That should do it! You can now deploy to a server that is sans SVN!

Caveats: Because of the way the SVN module and WGET work, I’ve had to not include he downloading of index.html (Basically WGET treats the directory listing as a page, and will output it as index.html) so this technique will not work if you have any pages called index.html in your structure. Work around: Rename all instances of index.html to index.htm

You might get some weird results if some one checks in code at the same time as you do a deploy - unless you have a bucket load of developers working on your system and you have no communication between developers, this is pretty unlikely.

(Names have been changed to protect the innocent)

Make scheduling jobs easier - use web services!

Posted on August 3rd, 2006 in cakephp, ruby on rails, work by Myles Eftos || 6 Comments

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

Web developers deal with scheduled jobs a lot. Any online application that deals with paid subscriptions needs to remind users when they need to pay up. Other apps may require data mining services to be run on a nightly basis.

This is pretty easy to do - run a cronjob under *nix or a scheduled task under windows. However, where things get tricky is when you have spent an in ordinate amount of time coding business logic into you application. Duplicating this logic for an external script to be run by cron is pretty silly, not to mention bug-prone. However there is a quick and simple solutions: Web services!

By setting up a web service that does your maintainence, you can leverage the classes and business rules that you have already written. For example, if we take a Model-View-Controller framework (such as Rails or CakePHP) we may have logic that will do data manipulations before the data is saved to the database. There is no way I would want to try and emulate what these frameworks do in an external script. By using web services, you are calling the framework natively and you avoid all of these problems.

How it works

It is really simple:

  1. Setup a webservice that will perform the functions you need to run on a regular basis.
  2. Create a simple script that calls the web services.
  3. Add this script to you crontab or task scheduler

The beauty is that you don’t even need to bother setting up the web service using SOAP or XML-RPC - REST will do the job quite nicely, especially since the web service is designed to only be used by you. However, when using rails I like to use SOAP, because it is so easy to setup and use. Here is a simple example:

housekeeping_controller.rb
  1. class HousekeepingController < ApplicationController
  2. wsdl_service_name ‘Housekeeping’
  3. web_service_api HouseKeepingAPI
  5. def maintain()
  6. # Run you maintainence script here
  7. end
  8. end

This file is the webservices controller. You would add your logic here - this may be expiring users or automatically checking email.

housekeeping_api.rb
  1. class HousekeepinApi < ActionWebService::API::Base
  2. api_method :maintain, :returns => [:string]
  3. end

This file defines the method to the web service caller. I usually set the return to type :string so I can output statistical messages or errors that occur during the job.

housekeeping_job.rb
  1. #!/usr/bin/ruby -w
  3. require ’soap/wsdlDriver’
  4. include SOAP
  6. begin
  7. wsdl = ‘path_to_wsdl_file’
  8. factory = WSDLDriverFactory.new(wsdl)
  9. housekeeping = factory.create_rpc_driver
  10. out = housekeeping.maintain
  12. put_s out
  13. rescue Exception
  14. $stderr.print “An error occured: #{$!}\n“
  15. $stderr.print “Detailes error description, if any:\n“
  16. raise
  17. end

This file is called by the cronjob. You will need to replace the string path_to_wsdl_file with the path to the real WSDL file. You get get the WSDL file from a Rails web service by querying /:controller/wsdl (so in this example http://www.yoursite.com/housekeeping/wsdl - pipe the returned xml to a file and save it.

Then it is just a matter of adding the housekeeping_run.rb command to you cronjob! Cron will even email you the results that get returned by your service - Nifty!

Nice an easy, eh?

Update: Improved validation in CakePHP for version 1.x

Posted on July 8th, 2006 in cakephp by Myles Eftos || No Comment

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

I recently started a new project in CakePHP and thought it time to use version 1. But it seems the way that the validator code works has changed, so here is the updated code:

The following goes in app_model.php

Updated validator code
  1. function invalidFields ($data = array())
  2. {
  3. if (empty($data)) {
  4. $data = $this->data;
  5. }
  7. if (!$this->beforeValidate()) {
  8. return false;
  9. }
  11. if (!isset($this->validate)) {
  12. return true;
  13. }
  16. if (!empty($data)) {
  17. $data = $data;
  18. } elseif (isset($this->data)) {
  19. $data = $this->data;
  20. }
  22. if (isset($data[$this->name])) {
  23. $data = $data[$this->name];
  24. }
  26. $errors = array();
  27. foreach($this->validate as $field_name => $validators) {
  28. foreach($validators as $validator) {
  29. if (isset($data[$field_name]) && !preg_match($validator[‘expression’], $data[$field_name])) {
  30. $errors[$field_name] = $validator[‘message’];
  31. }
  32. }
  33. }
  34. $this->validationErrors = $errors;
  35. return $errors;
  36. }

The validation helper method (See my previous post) looks like this:

  1. function tagErrorMessages() {
  2. $messages = “”;
  3. if(isset($this->validationErrors)) {
  4. foreach($this->validationErrors as $tag) {
  6. foreach($tag as $element => $message) {
  7. $messages .= sprintf(SHORT_ERROR_MESSAGE, empty($message) ? ‘Error in field: ‘ . $element : $message);
  8. }
  10. }
  11. }
  12. return $messages;
  13. }

3rd Degree e-news site publishes it’s first edition

Posted on March 24th, 2006 in cakephp, work by Myles Eftos || No Comment

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

The Edith Cowan University journalism students have launched the first edition of 3rd Degree - an online e-news site. The site was designed by Paul Bui and was developed by me in CakePHP.

The site allows the 3rd year students to understand the pressure of publishing a weekly news publication, with different teams controlling different parts of the process. If you would like to receive the weekly newsletter, you can register here.

3rd Degree is the brainchild of Kayt Davies, who is the lecturer in the unit.

New port80 event announced.

Posted on March 16th, 2006 in cakephp, work by Myles Eftos || No Comment

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

Port80 - the Australian Web Industry, of which I’m the membership officer has announced the next event - Ideas3.

The event features John Allsop from Sydney, who is a directory of Westciv, creators of Stylemaster; and Mark Boulton from the UK who is a noted typographer and designer who is currently working at the BBC.

For those of you who aren’t in Perth for the event on April 11, we will be posting the podcasts and photos after the event.

As an aside, the new Port80 site, which drives the event and memeber management systems was written by me in CakePHP.

Interested in starting a Port80 group in your local area? We are currently preparing an information pack. Port80 started as a way for web designers and developers in Perth could catch up in a casual, informal environment - usually at a pub. It is amazing how well it works - it’s a great way to meet other like minded people. You can read a better history on the Port80 website. Don’t forget to check out the port80 forums too!

« Previous Entries