Efficiency v's Robustness

I've been absent for a few weeks due to a trip to the UK to see my old mum and the rest of the family, followed by a couple of weeks of semi-deafness caused by having a head cold on a ten hour transatlantic flight and being unable to clear the fluid from my ears. It was both a blessing and an annoyance. It was a blessing as I work in a Dilbert like cubicle area, so the partial deafness acted like noise canceling earphones. However it was a major annoyance as I was constantly going around saying 'pardon, could you please repeat that.'

During my illness induced isolation I came across a learning episode for Project Managers. This related to yet another computer system melt down for the British bank, RBS. They had a system malfunction on Cyber Monday that lasted for quite a few hours during which their customers could not use their ATM, Debit, and Credit cards, or access their bank accounts. Many of their customers could not take advantage of the super deals on offer. More disconcerting was that they could not pay for purchases at supermarkets and petrol (gas) stations, or get any cash. Given that the bank handles 250,000 ATM type transactions per hour it meant that close to a million of their customers were seriously inconvenienced by this error.

This snafu on their computer system was a follow on from RBS's major melt down eighteen months ago when their customers couldn't access their accounts for days, in the case of their subsidiary Ulster bank the outage lasted for over a week for some customers.

The learning lesson is not that this bank has a less than stellar computer system, it is that it had lots of customers who where totally reliant on them for all their financial tools: Single bank account, single provider of debit and credit cards. The customers had no back up resources. All their eggs where in one basket, and it was a basket with a poor track record. Had these customers had accounts and cards with banks on different computer systems then they would have been inconvenienced, but not embarrassed. Standing by a petrol (gas) pump having filled up and not being able to pay is the epitome of embarrassment.

The lesson for project managers is that you need avoid all single points of failure that can bring your project to its knees. Even if that resource or asset has a good reliability record. There is no substitute for redundancy in a system when it comes to reliability. This lesson is particularly relevant in this age of cost cutting and efficiency. It is not commonly appreciated that the more efficient a system is the less robust it is. Efficient systems are by definition full of single points of failure, in fact the aim of creating the most efficient system is to remove all redundancy and make every element a single point of failure. In a choice between efficiency and robustness for critical systems we should pick robustness.