@madpilot makes

The second development tool in the “simple must have” category would be Capistrano. Capistrano is a deployment system that was written in ruby and is (not suprisingly) integrates quite nicely with Ruby on Rails. However! you can quite happily use it for any system, because at it’s heart, it is just a remote scripting tool.

When you run capistrano, you call a recipe, which is executed on the remote server or servers via SSH and it works really well.

Installation

The capistrano hand book has the latest installation instructions, so check there, but the basics (at the time of writing) are:

gem install capistrano

This will install the cap application. The tutorial from Simple Complexity – it deals with most of the difficult stuff (it’s actually really easy) and I don’t think I can put it any more eloquently.

Follow the steps and then you too can love the one-command deployment. I use it at MadPilot (for 88 Miles) and at Bam all the time. It’s great.

Next week: Part III

I’m sitting here watching the Poms getting destroyed by the Australians in the first test of the 2006/07 Ashes and just witnessed an interview with the man of the moment — Glenn McGrath. He just pulled a six wicket haul to knock off the fledgling English batting line-up in less that a day.

Anyway, during this interview he said the following when asked about what he thinks about when he is bowling:

It’s a simple game that we complicate. When you’re looking for that first wicket, your process is a little bit out

This sounds like the hype that is going around the web industry at the moment about keeping software simple – it is interesting to see how the theory translates to other facets of life.

The irony, I think, is that it takes much experience to be able to simplify a process down. An in-experienced person will probably see a simple solution, however they may fail to interpret potential problems down the track. semi-experienced people may not have used certain techniques long enough to be able to safely say in a given situation whether the procedure should be dropped or not. It is the experienced person that can see exactly what can be culled.

As a web developer, there are a number of tools in my toolbox that I find it hard to live with out. These tools for the basis of my development and deployment procedures and have not only made my life easier on many occasions, but saved my arse on more times than I care to admit.

Although these tools excel when used by a team of developers and designers, I still use then when I’m developing my own projects, such as 88 Miles. I’ve split this mini tutorial up into three parts, so that I can be a little bit more descriptive about the actual process.

Application #1: Subversion

Developing without subversion (SVN) is like throwing rocks at your house. Sure it may be fun, but sooner of later you’ll hit a window and lose all of your source code… I’ll leave the unmixing of that metaphor as an exercise for the reader. Subversion is a source code versioning system which you can think of as a safe for your source code. The concept is simple:

1. You take a copy of the source code (checkout) and you work on it.
2. When you have made your change you save the change back in to the repository(commit)
3. When you delete or stuff something up, you check the previous version out of the repository and no one is any the wiser.

SVN uses a copy-merge system which allows many people to check code out and work on it at the same time. This is handy if you have a lot of developers. Many other version control systems use the exclusive-lock in model, where a developer checkouts the file, which locks it until they are finished, which is fine too – both have their pros and cons.

Copy-merge pros:

• You can have many developers working on the same code at the same time
• If a developer leaves a project, goes on holiday falls in to a mighty cravass, there is no risk that they will have an exclusive lock on a file, stopping other developers working on it.

Copy-merge cons:

• This can get messy if two developers work on the same file at the same time. At check in time, they will need to merge their changes, and if they have been making major changes, this might not be trivial. Solving the issue requires some heavy communication (not really a bad thing) and some planning (again, not a bad thing). On small projects, the likely hood of this is pretty small

Exclusive-lock pros:

• It is impossible for two developers to work on the same file at the same time, eliminating a situation where one developer destroys another developers work

Exclusive-lock cons:

• It is impossible for two developers to work on the same file at the same time :)
• See Copy-merge pros

One of the greatest things about SVN is it’s price tag – nothing! (Don’t let that freak you out though, there are many big projects that use it – Ruby on Rails is one that comes to mind) and there are clients and servers for Windows, Mac and *nix.

Although setting up SVN itself is probably beyond the scope of this article, I’ll out line the basic procedure to set up a repository, and to work with that repository.

Setup

There is a command line utility bundled with the package called svnadmin. This, funnily enough, is the administration tool for subversion. To create a repository, you do the following:

svnadmin create /path/to/repository

Where /path/to/repository is the location of the repository. So, say you want to create project called foo in the directory /home/subversion, you would execute

svnadmin create /home/subversion/foo

And it is setup! You will probably have to play with the permissions by editing the /home/subversion/foo/conf/svnserve.conf file, for the sake of this excercise, we will allow unlimited access to the world. Obviously, in real life, you would lock it down so only those that you wanted to be able to access the code could. Check out the subversion manual for details on how to do this.

Open the svnserve.conf file and find the line that says:

# anon-access = read

Change it to

anon-access = write

SECOND WARNING: THIS IS INSECURE. DON’T DO THIS FOR A SERIOUS PROJECT. And if you do, don’t come looking for me when some one steals your code.

Now you can checkout the empty repository in to your working directory – so called because it is the directory that you work in. The command will probably look something like this (depending on our server setup of course)

Right! now you are read to start coding! Once you have finished, you can add any new files, delete any old file and then commit the changes. Just so I know what is going on, I will first check the status of the directory

A new_file_2

U modified_file

D deleted_file

The letter before the filename gives you a hint as to what you need to do – A is add, D is delete, U is modified. To complete this commit, I would run the following commands:

/path/to/working/directory% svn delete deleted_file

/path/to/working/directory% svn commit -m “Created some files, deleted a file, modified a file”

And if all goes well, subversion will return the new revision number. Revision numbers in SVN are global, so every time you run the commit call, the number will be incremented. The -m flag stands for message, if you don’t put it in, a text editor will pop up asking you to add a message. The message you enter needs to be descriptive of what you do, so that others (or yourself) can go back and see why each commit was done.

Want to rollback the changes from a commit? Say you made a BIG mistake in revision number 5 and you need to go back to revision number 4, running the following command should do the trick:

_/path/to/working/directory% svn merge -r 5:4 svn://svn.myserver.com/foo

/path/to/working/directory% svn commit -m “Undoing BIG mistake”

So revision 6 will now be the same as revision 4. Notice though that revision 5 still exists, so if you later realise that the mistake wasn’t that big after all, you can revert back to revision 5!

This is just touching the tip of what subversion can do – you can do all sorts of other funky stuff, but I’ll leave that up to you to read up on. 99% of the time, this is what a subversion session looks like for me.

Next: Trac bug tracking

I have just launched a new web site called 88 Miles. I was sick and tired of trying to juggle excel spreadsheets to maintain my client timesheets – I ended up having to double input (Because it was too much of a pain to enter the time directly, so I would write them down first) and would usually leave the task until the end of the week. Not suprisingly this part of my week wasn’t my favourite.

Enter 88 Miles. This little system that I have designed is meant to be REALLY basic. There is no invoiceing system or task system and this is by design. I wanted to make it’s primary role – that is entering time – extremely quick.

When you go to the site, you click punch in to the project you are starting. When you have finished the job, click the corresponding punch out button. That’s it. Really.

At the moment, the system is in beta so it is missing some of the other features that I plan implement before the official launch (Aiming for 1 June 2006) like RSS, an API and some desktop applets, but please feel free to go and have a play with it and let me know what you think!

As a bonus, those of you that sign up before the 1st of June get the first two months of access for free if they decided to continue after that point!

Prices look like they will be $24/year for the premium account – standard accounts will be free but limited to three companies and five tasks at any one time.

The tech:

The system is built in Ruby on Rails and is running on Textdrive.

Over at Port 80, we recently added a Ruby on Rails forum. This was met with both excitement and doubt from forum members. There were those that were excited about a new framework that promises to reduce development time by taking the tedium out of development. There were those who were doubtful about a system that has had a lot of hype, but hasn’t had the market infiltration to match it. Many people think that anyting that rises so quickly will only fall just as quickly. Will this happen to rails? No idea – only time can give us the answer.

However, I think there is an underlying factor here that has been over looked. The idea of frameworks designed specifically to speed up web development is what we should be getting excited about.

Web development, and to a degree, software development has changed. Customers are more than ever expecting cheap software that is good. It doesn’t make sense for us as developers to waste our precious time re-implementing the same parts of a system for different jobs. Introduction of frameworks that do this simply can only be a good thing.

One hurdle for many professional coders is the lead-time required for learning a new framework. Be the very definition, frameworks expect the coder to do things in a certain way. This may put many coders off, because they have to spend time learning, not coding and this costs money. What they may not realise, though, is that every hour they spend re-coding form validators instead of learning a framework that does it for them, adds up over time.

The old adage of work smarter not harder is extremely relevent here. There will never be more hours in a day. Trying to use all of them is impossible. Trying to improve productivity with out changes in thinking won’t work. We as web programmers know how this web thing works. The designers of these frameworks know how this web thing works. We really should be working together to work smarter.

An article was recently posted on the Wall Street Journal website entitled “WSJ.com – For Some Technology Companies, ‘Beta’ Becomes a Long-Term Label” which asked the question how can companies get away with leaving software in a “beta” state for so long. Google is notorious for doing it – Gmail has been around for at least two years and is still tagged as beta. In fact, many people are using Gmail as their primary email address. There are many real-world analogies as to why you wouldn’t do this in the article, so I won’t bother repeating them here.

It does bring up an interesting point though. How come web software can get away with, nay, thrive on, releasing beta software?

Traditionally, the main thrust of software was the underlying business logic. It has only been in recent times where user interfaces and user experience has become important. It is often impossible to know exactly how a user is going to react when they start to use an interface. Being in a constant state of beta allows designers to be constantly tweaking elements of the design. If things change drastically, users will put it down to the fact that the site is still in beta.

This is relevent when adding features. As a developer, you may find a new “must have” feature that you are sure that your web site requires. It is a more effective use of your time to build a version quickly, so that you can gauge it’s popularity, and then concentrate on making it robust if you know it is worth-your-while. You couldn’t get away with this without being in a beta state.

Obviously, this isn’t a mechanism for everyone. I would even think about using a security product that was in beta. But for web apps that aren’t doing anything mission critical then it should be fine.

I recently read about the deployment model that Flickr uses. After picking my self up from the floor, it made me think about my own deployment methods, and I have started implementing a new system, which I though I would share.

First off all, let me describe the types of projects that I have to deploy:

1. Internal MadPilot or Personal projects – i.e. stuff hosted on the server at my house (which also doubles as my development server)
2. External projects that I can take a local copy of to work on
3. External projects that I am forced to work on “off-site” i.e on the clients server.

The deployment system I will describe here covers project type 1 and 2. Not much I can really do with 3, as I have no control over other people’s servers…

Most of the projects I work on, I’m the only developer, but the system described is designed to scale to multiple developers.

Step 1: Set up a subversion repository

I have my subversion repository structured as such: [svn_root]/MadPilot/Clients/[client_name]/[job_name]/

Because I do a lot of outsourced work, I use the job_name to denote different jobs for the same client. If the client is a one off (i.e. the client isn’t another web company) I will usually leave this off. The website code will be stored in the Website directory. This allows me to store documentation and other bits and bobs outside the development tree.

Step 2: Checkout the empty tree

Next I create development and test directories with in my webserver: usually of the form [web_root]/clients/[client_name]/[job_name]/dev/[developer_name] and /clients/[client_name]/[job_name]/test.

I then check out the empty subversion tree to all of the directories so that they become under source control.

Step 3: Setup the webserver

At this point I will set up a virtual host for the client of the form [developer].dev.[job_name].[client_name].clients.madpilot.com.au and test.[job_name].[client_name].clients.madpilot.com.au (Yes, this does end up with stupidly long URLS, but I prefer them to be descriptive. No one sees them other than the client anyway…)

Step 4: Import the initial tree

It is at this point I download the existing site, or start a fresh if it is a from-scratch job. I then add all the files to the repository.

Step 5: Setup any databases

I try to make sure that every developer has a separate database, so does the test system. Usually named [client_name_[job_name]_[development_name]

Using the system

When you start working on a project, you make sure you login to your working directory on the server (usually via SSH) and do a subversion check out. When you finish, you check your work back in. When the system is at the point of testing by the client, you check everything out to the test site.

Using subversion means that you have a versioned backup of all the code, and you can manage multiple developers and a test system at the same time.

Going to production

This can be the tricky bit. For projects on my server, I usually just do an export into the production directory. If the system was tested well enough, if should just work. Gotchas include paths to local files and permissions, but this should be documented anyway.

On external systems, this can be a little more difficult. At the moment, I export the code into a temporary directory, and manually FTP the files up. I hope to be able to automate this in some way, even if it will just reduce me typing.

Next thing to do is to write a nice web front end to subversion so I can do “one-click” deployment… Ahhh :)

So that is my plan – what does everyone else do?

I quick entry today, as I really should be working.

I came across a PHP MVC (Model/View/Controller) framework for PHP 5 – You can think of it as the Rails bit of Ruby on Rails. It is this sort of stuff that will continue to push PHP into the spotlight and allow it to compete with the big boys…

It is called Agavi PHP MVC Framework

Web 2.0 is about data exchange and classification. Taking a high-level look at software design process one of areas that is good in theory, that fails in practice is data exchange and classification.

Think about the last software project you worked on – would a new programmer be able to pick up the documentation and work out what is going on? Could they be able to find the documentation? Was there documentation at all?

Documentation is all about data exchange – I bet if you went through your email or project mailing lists you could piece together a decent amount of usable documentation (Implementation decisions, solutions, bug reports etc) – Can Web 2.0 provide the glue to ease the burden of documentation?

Imagine being able to tag bugs and design decisions so searching for a bug returns a link that points to the design documents and comments entered at check-in. This gives the developer background information quickly – and these docs are living and more easily maintainable. As a developer, you don’t have to change to documentation mode – it is already part of your work flow (Other than remembering to enter check-in comments – but you already do that don’t you?)

We don’t necessarly have to limit this to documentation either. Project managers could use it to gauge where the project is at, clients could actually report bugs with out having to learn an obscure bug reporting interface or software terminology. How about timesheets? Add an entry stating you started work at a particular time, and then enter another entry saying you finish at a particular time – tag it wit the project name, maybe what part of the project you were working on – Voila! Or meeting minutes – these often have action items interleaved – tag them and make them searchable. Tag them as completed when you are done.

But I think the best bit is that it wouldn’t require anything more than an email client or a blog-like web interface. EVERYTHING IN ONE PLACE! I know I personally hate having to log in to a different apps for bugs, and meeting minutes etc. We should be letting the software organise our data, and leveraging search.

I think I will be revisiting this one soon…

I was going thorugh the posts from my old, now-defunct blog, seeing if there was anything I could bring over here — it is amazing how much can change in a year. There was an article I wrote about over use of cool techniques. In that article, I made mention to some new fangled technique called “Design Patterns”. At that point, I had no idea what they were and frankly couldn’t care.

Well, after being forced to look at them more closely for a uni assignment, I’m kind of a little hooked.

Design patterns are abstract solutions to common problems. Huh? Yeah, that is what I said. Many programmers strive towards code re-use. Design Patterns encourage thought re-use. Why re-invent the wheel? And because they are abstract (i.e. no code), they can be “ported” to different languages easily.

The Gang of Four introduced 23 of the things. They have put the challenge out for the discovery of more, and they haven’t been too successful as it is believed that almost all problems can be broken down to fit a composition of these rules.

To implement many of these design patterns correctly, you really need OOP features such as Abstraction and Interfacing. As I have pointed out many times, PHP4 doesn’t have these. PHP5 does. However! if you think about it, you can still use design patterns in PHP4 — you just need to be a little bit careful.

I won’t go through ever design pattern just now, but, I’ll outline one, and maybe add to them in the future. [By the way — an awesome book to use for getting your head around design patterns is “Head First Design Patterns” — look it up on Amazon]

The Stategy Pattern

Programmers often get into the habit of extending classes to change functionality — maybe because it is the easiest OOP function to understand. It can lead to problems though when you need to change implementations in a base class. I’ll use an example from a system I was writing the other day: the ubiquitous PHP email sender. The job I was doing required two types of email to be sent — a run-of-the-mill text email and a text email with an attachment.

Because I’m always trying to build my code libraries up, I decided to create set of classes that will do the job. Now, pre-design patterns, something like this may have happened:

Nothing wrong so far, but what happens if we want to add a HTML email? We could probably add another class below the attachments, because the HTML component is an attachment I guess, so we add the HTML file as one of the attachments at design time. This isn’t a great solution though, because one of the necessary bits to a HTML email is the HTML content, and you could theoretically remove it. How about we add another variable specifically for the HTML copy? That would also work, but what if the client decided that they wanted to select between HTML and text emails dynamically at run time? You would end up with something like this code wise:

switch($mailType) {  
  case “text”:  
    $email->setBody($text);  
  break;  
  case “html”:  
    $email->setHTMLBody($htmlText);  
  break;  
}

(This is a bit of a contrived example, so run with me here)

now what happens if you want to add another type of email? You have to modify the case statement and test the whole thing all over again. Everyone hates testing – especially code that you know used to work that you had to change. Two keywords: CHANGE and TESTING. Minimise both and we as programmers are happy.

Now my solution was to create a class that has everything an email needs – To, From , Subject and a repository for headers (Plus a few other bits and pieces, which I won’t mention to reduce clutter). It also has a function called send(). Wow, ground breaking. Where the design gets a little strange, is there is another variable, called $emailType. This variable stores reference to a class of type EmailType (Well it pretends to, PHP4 doesn’t support interfaces). So, any class that implements EmailType can be stored in that variable. One of the abstract classes (again, let’s pretend it is abstract, PHP4 won’t understand) is the createMessage() function. This is where the magic occurs…

Each class that implements that interface know exaclty how it message needs to be contructed. The base email class doesn’t care – as long as it gets a string to tack on to the email it is happy. The creation of messages is de-coupled, meaning you can create a new email class without changing any exisiting code (As long as it implements the interface correctly).

This is an implementation of the strategy pattern!

Formal Description [Ganf of Four]: The strategy pattern defines a family of algorithms, encapsulates each one, and makes them interchangable. Strategy lets the algorithm vary independently from clients that use it.

Which is what we just did… More on design patterns later.