@madpilot makes

You know that feeling you get when something you’ve been taught to believe in gets discredited and because your belief was tenuous at best, the walls come tumbling down around you and then you have a huge weight lifted off you shoulders?

Pascal just posted this on the 220 mailing list. Amen. It’s something that I’m pretty sure I’ve been articulating for a long time. Whenever someone has asked me why software is hard, I always use this analogy:

If you ask a Civil Engineer to build you a bridge, it is easy to spec out. You know how far the bridge has to span, what sort of foundations you need, and as a result you can make a recommendation about what sort of bridge you need. The Engineer can build you a little model – you can look at the model and say “Yes! That is a bridge. That will do nicely”. They can mathematically model the bridge to make sure this doesn’t happen. They build the bridge and if it allows things to cross from one bank to the other, you have a success.

Unless you are building “Hello World”, a Software Engineer’s life isn’t so simple. You have different platforms, users, stakeholders, contexts – it gets exponentially harder with every feature that gets added. I once did a unit at Uni called Formal Methods which tried to mathematically model software. It was stupid. The code we modelled was like, nine lines long, and required a 32 page proof (I didn’t even get close). Stupid.

Of course, academics have been trying to shoehorn software into engineering for ever. In first year, they taught us UML which I guess is similar to architectural drawings or flow diagrams or something. I’m sure UML works really well when working with the waterfall model of software design, which has strong ties to old school, proper engineering. I couldn’t imagine having to go and update hundreds of UML documents every time a minor change was required. We are also taught in first year, that the waterfall model is pants in the real world, which by association makes UML nothing more than a nice thought experiment. (I’m still bemused by the number of Software firms that put it as a requirement for graduate Software Engineers – basically because coming up with job descriptions for inexperienced programmers is really hard).

Sure you can argue that testing is a software technique that we (should) use, but this is the exception to the rule. I guess the conclusion we need to come to is that Software isn’t an engineering problem – it’s a people problem. (Some may say, it’s a creative problem – that’s also true, but buy me a beer and I’ll explain that traditional engineering is too, so the argument doesn’t further my point). This in itself is a problem, as (gross generalisation ahead) boffins who like coding, tend not to deal with real people very well.

Further discussion on our internal list suggested that creating software products is the way to go. I think I want to agree with this – there are many examples of off-the-shelf products that are extremely popular: Microsoft Office, Adobe Photoshop etc. In these situations, the customer works with in the workflow of the software, and that seems to work. So do we as developers need to convince our clients that the feature they want may not be needed? Do our clients actually know what they need? Of course this view is not with out it’s flaws either – users will generally be working against the software, rather than with it. Is working with a sub-optimal solution better than battling with requirements and budget overruns?

I can’t help to think that there is something we are missing. It would seem there is a disconnect between what our clients want and what we can provide. If you look at the classic project triangle, your client wants to minimise price and time, and maximise good (I hope my English teacher isn’t reading this), where as we want to maximise all three. So the crucial “pick two” part flies out the window. Either we start sacrificing the good, re-negotiate the price, or try to stretch out the project to restore the balance – none of which makes for happy clients.

Well how about adding fat to the quote? In theory, this is fine – if a client sees value in an “inflated” (but more likey a realistic) price then everyone is happy right? Well, not really – software development is much like homework assignments: You start out with plenty of time, and the best intentions, and then end up pulling an all-nighter to get it finished – and you still only get a C at best. I suspect this is because it’s impossible to lock down requirements of an abstract problem. This isn’t only because of the difficulty in describing what we don’t understand, but because we don’t even know what half of the problems are going to be.

And this is our quandry – how can we estimate unknowns? Not just “we haven’t seen this before but it looks like X” unknowns, but “What the hell? How is that even possible?!” unknowns. Other areas of Engineering encounter these problems occasionally – we get them all the time. So, the solution (he says as if there is one) is to minimise the risks and/or consequences of these unknowns. Jobs that deal with people do this all the time. If you work in marketing, you can postulate all you like – you can’t be sure how a campaign will work until it does. Marketing is reactive.

When you make a change you can’t be sure what will happen. Sure, you can put an ad in the Yellow Pages year after year, because it has brought in on average Y leads per year – but there is no guarantee this year will be the same. It seems that the humanity-based sciences are happy with this, but quantitative-loving geeks don’t like that. Hell, binary is black and white, not Gray.

So, perhaps the key is to treat software as a living breathing thing. Agile programming and iterative development can help, but they are means to an end – they don’t work with out communication and understanding between people. We need to break down the barriers between provider and client – the question is: Is that even possible?