Bloggy Hell

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

You know that feeling you get when something you’ve been taught to believe in gets discredited and because your belief was tenuous at best, the walls come tumbling down around you and then you have a huge weight lifted off you shoulders?

Pascal just posted this on the 220 mailing list. Amen. It’s something that I’m pretty sure I’ve been articulating for a long time. Whenever someone has asked me why software is hard, I always use this analogy:

If you ask a Civil Engineer to build you a bridge, it is easy to spec out. You know how far the bridge has to span, what sort of foundations you need, and as a result you can make a recommendation about what sort of bridge you need. The Engineer can build you a little model - you can look at the model and say “Yes! That is a bridge. That will do nicely”. They can mathematically model the bridge to make sure this doesn’t happen. They build the bridge and if it allows things to cross from one bank to the other, you have a success.

Unless you are building “Hello World”, a Software Engineer’s life isn’t so simple. You have different platforms, users, stakeholders, contexts - it gets exponentially harder with every feature that gets added. I once did a unit at Uni called Formal Methods which tried to mathematically model software. It was stupid. The code we modelled was like, nine lines long, and required a 32 page proof (I didn’t even get close). Stupid.

Of course, academics have been trying to shoehorn software into engineering for ever. In first year, they taught us UML which I guess is similar to architectural drawings or flow diagrams or something. I’m sure UML works really well when working with the waterfall model of software design, which has strong ties to old school, proper engineering. I couldn’t imagine having to go and update hundreds of UML documents every time a minor change was required. We are also taught in first year, that the waterfall model is pants in the real world, which by association makes UML nothing more than a nice thought experiment. (I’m still bemused by the number of Software firms that put it as a requirement for graduate Software Engineers - basically because coming up with job descriptions for inexperienced programmers is really hard).

Sure you can argue that testing is a software technique that we (should) use, but this is the exception to the rule. I guess the conclusion we need to come to is that Software isn’t an engineering problem - it’s a people problem. (Some may say, it’s a creative problem - that’s also true, but buy me a beer and I’ll explain that traditional engineering is too, so the argument doesn’t further my point). This in itself is a problem, as (gross generalisation ahead) boffins who like coding, tend not to deal with real people very well.

Further discussion on our internal list suggested that creating software products is the way to go. I think I want to agree with this - there are many examples of off-the-shelf products that are extremely popular: Microsoft Office, Adobe Photoshop etc. In these situations, the customer works with in the workflow of the software, and that seems to work. So do we as developers need to convince our clients that the feature they want may not be needed? Do our clients actually know what they need? Of course this view is not with out it’s flaws either - users will generally be working against the software, rather than with it. Is working with a sub-optimal solution better than battling with requirements and budget overruns?

I can’t help to think that there is something we are missing. It would seem there is a disconnect between what our clients want and what we can provide. If you look at the classic project triangle, your client wants to minimise price and time, and maximise good (I hope my English teacher isn’t reading this), where as we want to maximise all three. So the crucial “pick two” part flies out the window. Either we start sacrificing the good, re-negotiate the price, or try to stretch out the project to restore the balance - none of which makes for happy clients.

Well how about adding fat to the quote? In theory, this is fine - if a client sees value in an “inflated” (but more likey a realistic) price then everyone is happy right? Well, not really - software development is much like homework assignments: You start out with plenty of time, and the best intentions, and then end up pulling an all-nighter to get it finished - and you still only get a C at best. I suspect this is because it’s impossible to lock down requirements of an abstract problem. This isn’t only because of the difficulty in describing what we don’t understand, but because we don’t even know what half of the problems are going to be.

And this is our quandry - how can we estimate unknowns? Not just “we haven’t seen this before but it looks like X” unknowns, but “What the hell? How is that even possible?!” unknowns. Other areas of Engineering encounter these problems occasionally - we get them all the time. So, the solution (he says as if there is one) is to minimise the risks and/or consequences of these unknowns. Jobs that deal with people do this all the time. If you work in marketing, you can postulate all you like - you can’t be sure how a campaign will work until it does. Marketing is reactive.

When you make a change you can’t be sure what will happen. Sure, you can put an ad in the Yellow Pages year after year, because it has brought in on average Y leads per year - but there is no guarantee this year will be the same. It seems that the humanity-based sciences are happy with this, but quantitative-loving geeks don’t like that. Hell, binary is black and white, not Gray.

So, perhaps the key is to treat software as a living breathing thing. Agile programming and iterative development can help, but they are means to an end - they don’t work with out communication and understanding between people. We need to break down the barriers between provider and client - the question is: Is that even possible?

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

As anyone who has ever received the dreaded Object doesn’t support this property or method error in IE can attest, debugging using everyone favourite browser is a right, royal pain in the heiny. Using Firebug in Firefox really has spoilt us as frontend developers (Hey, what am I talking? These ARE BASIC TOOLS that every other development platform has had since Ada Lovelace was in small britches, but I digress), so what is a cross-platform developer to do?

Little known fact outside of the .NET world: Visual Studio has a complete JavaScript debugger built in, which allows you to set breakpoints, add watches, mess around with variable values and more importantly, gives you better error messages, and actually highlights the line where things went wrong.

1. Download and Install Visual Studio Express Edition (which is free as in beer).
2. Create a new Web Site - call it what ever you like, it’s just a placeholder
3. Hit “Run” (or F5) on the blank problem - a local server should start, and IE7 should load a blank page
4. Change the URL to the page you want to debug. Once the page is loaded the debugger is good to go - in fact, if your page has any errors, Visual Studio will get focus, and politely tell you as such
5. To add a break point, flick back to Visual Studio Express and open the JavaScript file you wish to add the break point to, and then refresh the browser - once context hits the point, you will be able to step through the code.

Whilst still lacking a DOM browser (Firebug Lite might be able to help out with that), this takes some of the fun out of debugging JavaScript IE, from the point of view that it is now actually possible.

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

If you are a DBA, and your reading this - look away now, because I’m pretty sure they covered this in Database Optimisation 101 and you WILL laugh at me having this revelation. 88 Miles hasn’t been the snappiest web application around lately thanks mainly to an influx of users (NOT that I’m complaining :P). I’d successfully added some views to speed up some of the reporting recently, and I went through today and optimised a lot of code, but it still wasn’t as quick as I would have liked it (A page load in the main index page was taking on average 1.5 seconds - down from the 4 seconds pre-optimisations).

I was googling the performance differences between INNER and LEFT joins (INNER wins most of the time for those of you playing at home), and came across a word that I vaguley remembered between dozing off in my Database class at university - indexes. Now, don’t get me wrong, I KNEW these things existed, I even knew what they did, but because I don’t use them regularly, I didn’t even think to look at them. As all of the primary IDswere already primary keys, my gaze turned to the foreign keys (I use the term relatively loosely - they were foreign keys in the sense that they referred to another table ID, not because they had been explicity setup that way).

I added indexes to the foreign keys on the three main tables, and voila! A ~10x speed increase on that front page! It’s such a simple optimisation too! *Slaps head*

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

As any website grows, we often find ourselves having to move servers to deal with capacity, or better prices or whatever. This can be a right pain to do, but with a little planning and a few tricks, you can make the process somewhat smoother. In this quick howto, I’ll cover how to move a typical web setup: Web server, DNS, Mail and database. Of course there are more than one way to skin a cat, but this method works quite well if you have a couple of days up your sleeve and has the advantage of only costing you a few minutes in downtime. In this example, we will pretend to move http://www.mycoolwebapp.com from the IP address 192.168.0.1 to the new IP address of 10.0.0.1 (Fake web host, and fake, local IP addresses - substitute with real values).

Step 1. Move your DNS

DNS can be the most painful step, as it can take up to 72 hours to move from one primary DNS server to another. In a nutshell, DNS acts like a big phone book, which tells your browser that http://www.mycoolwebapp.com belongs to the IP address 192.168.0.1. When you enter http://www.mycoolwebapp.com in to your browser, it will ask the operating system for the IP address. If the Operating system has it cached, it will return it. If the cached value looks old (ie the TTL has expired), or it doesn’t know about it it will ask a parent DNS server (or root server) where it can find the updated records, and will then fetch the new records from the relevant DNS server. The TTL is usually set pretty high - in the range of days, as generally Name-to-IP address mappings don’t change much, but if you do change, there means there could be a couple of days before all the servers around the world are updated, meaning your site won’t be found during that time!

Most users don’t have control of the Time-to-live (TTL) value on the root servers, so you need to ensure that both your old primary DNS server, and your new primary DNS server mirror each other. This way, regardless of whether the user get old information or new information, they will still be directed to the same information.

Most users WILL have control of the TTL on their own servers, so we can set that to something small which will make switch over much easier. Make note of the TTL on your old DNS server - you will need to wait that long after changes things before you can switch over to your new server. Setup your new DNS server to point to the old server (192.168.0.1) - ensure that all of the values are exactly the same. Now, on both servers, set the TTL to something very small, say 10 minutes (which is 600 seconds).

As this can take a couple of days to for the root DNS changes to propagate, you may as well change the root DNS details now. This will usually done via the domain registrars control panel. Simply change the primary and secondary DNS server IPs from the old DNS servers to the new ones (192.168.0.1 to 10.0.0.1 in this example). Make sure you double check all the values, because if you make a mistake it can take days to rectify!

Step 2. Setup the new webserver

Now that the DNS is setup and in the process of re-delegating, you can setup website on the new server for final acceptance testing. This is usually the easiest part of the process. What I do here is deploy the current software to the new server, take a snapshot of the database (phpmyadmin helps here if you are on MySQL, but each database system has a mechanism for backing up the database) and copy over any uploaded files etc. Now, most shared hosts will use virtual hosts, which means it serves up different pages based on the domainname, not the IP address, and if your software relies on the domain name for functionality, it can be really hard to test all the functionality with out the having the name-to-IP address mapping in place (remember, at this point http://www.mycoolwebapp.com is still pointing to 192.168.0.1).

Hosts file to the rescue! You might not know this, but all the major operating systems have a “hosts” file, which is checked before a DNS check is made, so if the operating system finds the requested domain name in that file, it won’t even bother querying the DNS server. By associating the new server IP address with the domain name in this file, we can actually view what is going on on the new server (Just don’t forget to delete the entry when you are done!). Windows users can find the file in c:\windows\system32\drivers\etc\hosts, and Linux (and I’m pretty sure OSX users) can find it in /etc/hosts. There is usually examples in the files, so it’s best to follow those, but I know these values work for Windows and linux:

10.0.0.1    www.mycoolwebapp.com

You may need to restart your browser for the change to take effect. Now if you go to http://www.mycoolwebapp.com you will be taken to the new server allowing you to set it up and check everything is working properly with out affecting the currently live version.

Step 3: Setup mail

Email can be pretty painful to setup, and is one of those things that will get you in a lot of trouble if you stuff it up. First off all you need to know all of the email addresses associated with the domain. If your hosting provider uses a web admin control panel like Plesk, this is usually pretty easy. Mirror all of the accounts on the new server, and make sure all of the quotas are either equal to or greater than the current values. If you need to setup new passwords for all of the accounts, note them down, as you will need to notify each user of their new password. Don’t forget to check for things like aliases and forwards.

Step 4: Check and double check

Make sure you note down any other bits and pieces that may have been setup, like cronjobs, other services and make sure that things like mail contact forms etc actually work. There is a trick here for young players with email forms - the resulting email may go to the new server, not the old one, so if you test a contact for and no email appears, check the new server.

Step 5: The big move

The way I usually play this is to replace the website on the old server with a “We are down for maintenance” page. This has the advantage of ensuring no one posts any new information whilst the the site is in flux and also gives you a visual confirmation that things worked. If your server can use .htaccess file, this is pretty easy using a rewrite directive. If the framework you use has friendly URLS, there will probably already be a .htaccess file that tells the server to rewrite the URLS to file - this is the simplest solution - just create a maintenance.html file and change the rewrite to look like this:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ maintenance.html [QSA,L]

This allows you to still access CSS, images and JS files, so you can style the maintenance page, but any request to a file that doesn’t really exists will get piped through to the maintenance.html file.

If .htaccess isn’t an option or you have a lot of static file, you might have to do the old fashioned “move the current site root and replace with a new one” trick (There are rewrite rules you can do, but this just might be easier). Create a new directory that has the maintenance file (You would probably have to call it index.html in this instance) and put any associated images, CSS and JS files. Then move the current site root (for example public_html) to say, public_html.old and then move the new directory to replace the old one (ie public_html). When you view the site on the old server (you might need to remove the entry in the hosts file to see it) you should now see the maintenance message.

Next, re-sync the database and any new uploaded files and give the new server a final test (by putting the hosts file entry back in). Once you are happy everything is working, we can flick the DNS over.

Step 6. Flicking the DNS

On both the old and new server, change the IP addresses for all of the relevant entries from the old IP address to the new one. Within 10 minutes, everyone should be seeing the new server’s version of the site! This also means that all of the email will be heading to the new server to, so…

Step 7. Re-configure mail clients and syncing up folders

If you configured new passwords for all of the mail clients, they will need to enter the new password into their mail clients. I generally walk them through creating a NEW mail account on their clients - the reason for this will be revealed in a second.

If they have been using POP in the past, then they will probably already have local copies of their email, so your job here is done. However, if they are using IMAP (probably more likely now) you will need to sync up the two accounts. There are tools out there that can do it, but you will need to know the passwords on both servers, and they can be a bit hit and miss - thankfully you can use the users mail client to do the sync.

Simply create a new account on their mail client that points at the new server, and change the old account to point to the IP address of the old server. If done correctly, the user should now be able to access their old email and folders as well as any new email on the new server. Most mail clients will actually allow you to drag folders across servers - do that and your email will be synced up! (It might take a while, if they have a lot of archived mail). After that is done, you can delete the old account and set the new account to default.

Step 8. Shutdown the old server

The final task is to shutdown the old server. I usually leave it running for a week to be on the safe side, mainly to recover any email that the user forgot to move across (It ALWAYS happens).

Step 9. Do and get a drink

You deserve it - it’s probably been a big week!

I hope that this made sense, or at least acts as a resource when you try to explain to a client why it can take a week to move servers! Of course, these instructions are pretty general, so your mileage may vary. Golden rule is don’t rush and double check everything!

TwitThis || StumbleUpon || Digg it || del.icio.us || Reddit

There comes a moment in every career where you realise that there is a whole world outside of what you do. Sure, you don’t have to have three PhDs to figure out the world of macrame is significantly different to Ruby on Rails hacking, but when was the last time you thought about process in, say banking software development? Or had a look at what is going on in the world of Operating System code? To an outsider, these are related industries - they both involve sitting in front a grey box for hours on end banging gibberish on to keys with embossed letters that are out of order, but to us, they are worlds apart.

A couple of days ago, something reminded me that I, and I assume a great number of other web developers have forgotten our brethren that are more at home with linked-lists than unordered-lists, and as a result, we are significantly reducing our ability to find the right tool for the right job.

This revelation was borne from the recent Twitter downtime. As always happens, hundreds of well-meaning developers tried to offer up solutions to fix Twitters yo-yo like uptime graph - there was the usual Rails bashing, some bewilderment over MySQL replication, and a general consensus that is shouldn’t be this hard, but one article really stood out: Scaling a Microblogging Service. Go and have a read if you want to know why scaling Twitter is REALLY HARD (maybe impossible),

For those of you who don’t mind a spoiler, the crux is that Twitter is designed on a platform more suited to content management, when really it is a messaging system.

Now, you go to any web developer on the planet and ask them to build you a Twitter-clone, and I bet you each design would be pretty similar. You would have a table for users, a table to hold friend references, and a table for messages, which would all be linked via some sort of foreign key relationship. The reason being is that for 90% of what we work on on a day to day basis, this makes the most sense. Generally, you have few authors, many readers and those readers all get basically the same information.

Twitter is is completely different. Everyone is a producer and everyone is a consumer. Every second there are hundreds if not THOUSANDS of writes from people leaving their 140 characters worth of thoughts. To make matters worse, every user get a different view, so every page load would be absolutely thrashing the Twitter database servers.

Twitter is a MESSAGING SYSTEM - not a CMS. It closer to a mailing list then a blog. So why did the developers not make this observation straight away? Well, as web developers, this is foreign territory. If Nokia decided that they were going to build Twitter, I bet things would look very different. Twitter and SMS are pretty closely related, and for someone dealing with short messages all day, that may have been the natural path for them to take.

You might counter this argument by saying that we are web developers - of course we think in terms of web. We do what we know, just like everyone else. Correct. But maybe we should stop and think a little before we dive in and start coding up prototypes using our favourite RAD tools. Maybe we should try to look for metaphors in other areas before taking our hammer to unsuspecting screws, bolts and watermelons.

Maybe we should take some lessons from the User Interface guys? They use metaphors all the time when they design. Without somebody realising that people like putting paper into manila folders, we would still be kicking around text terminals. Lucky for us, our metaphors don’t need to be as abstract as that. We only need to look at outside our little bubbles to see what developers from other industries are doing and we may too see a obvious solution that we would have otherwise missed.