Software management: Waterfall, Agile, and Kuhn

In 1962 Thomas Kuhn wrote his renowned book 'The Structure of Scientific Revolutions' which among other things introduced us to the "paradigm shift", a hackneyed phrase much beloved by dull managements these days, but a significant new theme at the time. 

Also in the book was the concept that the adherents of the old way of looking at the world, the old paradigm, could not, with their previous training and experience, understand the new way of looking at the world. This inability to comprehend he termed as incommensurable, a clumsy word hardly surprising it lost out to paradigm. One of the books findings was that the adherents of the old paradigm are not so much converted to the new, but rather they are submerged by it. They and their views die out.

So what has this to do with software management? A great deal is the answer. Currently we have a major debate within the software project world about whether we should use waterfall or agile planning and development methods. Much ink has been spilt, and many strong positions taken. Definitely there is a great deal of incommensurabling going on and the triumph of one over the other is not on the horizon, far from it.

The problem for we project managers is that we are often called upon to bridge the gap. To be translators at best, peace makers at worst, between the disputing theologians. In addition we also have to communicate our schedules to senior managers who are steeped in the point value world. "Give me a date when this will finish and the next phase will start" is their usual response to scheduling questions. Iterative development with its vagaries and imprecision is a foreign land to them. Oh, they get the concepts of the method but they can't break free of the point value mind set.

I had an example recently on a database project I'm managing. We are using an iterative approach with scheduled releases of the database. The later releases will be more a case of fine tuning the model rather than major upgrades. The development, testing, and deployment will continue until the main program is completed. The applications that will use the database understand the approach and when they'll be able to use the new code for their development effort.

So far so good, the techies understand what's going on. Now we come to the incommensurable part, the executives managing the entire program. They want everything on one chart -as an aside I think PowerPoint has a lot to answer for in the ongoing process of dumbing down corporate executives - everything has to have discrete start and end dates that line up with dependent activities, they want waterfall.  After days of futile discussion, basically talking past each other, I gave them a waterfall version. As they say in the espionage world it was a convenient fiction, all that was missing was the traditional starting and finishing phrases of children's fairy tales. Whether they'll all live happily ever after is debatable, it is a software project after all.

The political learning point of this example is that you need to understand the requirements of all of your audiences and accommodate them as well as you can. Leave them comfortable with their illusions while you manage your project in the most effective manner: Duplicitous yes, politic absolutely!

Self sabotage and the OODA loop

I never underestimate the capability of executives to sabotage their own plans in the rush to meet some arbitrary financial target. Their inability to think holistically is systemic, if not genetic. If they used the OODA loop they might avoid it.

Two recent examples, one international in scope, the other more mundane, will illustrate the propensity to self harm. In the UK the leading supermarket chain declared a few years ago that it would use a profit margin of 5.6% as it's guide for all company planning and that it would coordinate all it's activities to maintain this high margin. The value was sacrosanct, had to be met, no matter what.

Well, why they were preoccupied with meeting and maintaining this target they took their collective eye off the market. In OODA terms they stopped observing. So when the market changed as the economic austerity caused by the Great Banker Cock Up of the mid 2000's made shoppers both more frugal and discerning they failed to orient to the new reality. Very difficult to re-orient the business if you don't pay attention to changing conditions because you are focused, anchored as it were, on a meaningless target. Like a sailor following the North Star and not looking out for reefs and rocks.

This fixation resulted in them losing market share to low cost provider at frugal end of their customer spectrum and to quality suppliers at the discerning end of their customers.

The second example is in the area of American cost cutting. In many companies the start of the new financial year heralds the season of the annual resource action - euphemism for job cuts. Well in this company the IT department was targeted for some blood letting. At the time they were in the midst of an internal audit, a big deal in their company. The auditors had stipulated that the department must use a specific tool to ensure uniformity of process, thereby making the auditors life easier.

So it was somewhat surprising, but knowing the managers not shocking, when they fired the only guys who knew how to use the tool. So the audit came to a screeching halt and a a panic of the "hair on fire" type ensued. Talk about not thinking things through, of not orienting correctly for the given situation. They could have kept the guys on until the audit was over, or kept one of them. Obviously no holistic thinking taking place. Totally fixated on reducing head count by a certain date and oblivious to the consequences.

In both cases lack of observing and orienting skills led to self sabotage. Bad enough when people are working against you, but it is really takes the biscuit when you do it to yourself. 

Interfaces and interlocks

Whenever a project relies upon other projects to provide key inputs, so called dependencies, the risk goes up. When the dependency is a clear one it is usually well managed. The dependent project has a very keen interest in it being so. But where the dependency is more opaque, or an interim solution is required, things can get overlooked or a complacent attitude prevails. In both cases the problems are discovered late and the dependent project is thrown into a panic.

I'm currently working on a project that is part of a much larger program, with numerous work streams involving infrastructure, networking, database design, and application development. One of the key activities involves transmitting a large amount of data for testing purposes from a mainframe site to server farm for database development, I'm managing the database part. The files are large and in production will be carried over a dedicated network. The network has been delayed and won't be available for the early part of the development effort. So an alternative transmission connection was required. This is when the chances of the wheels coming off started to increase, because once the network delay was identified and the question of transmission was raised the answer was; don't worry we'll move it using a virtual private network (VPN), no problem.

The moment this glib statement was made then this cynical PM immediately went on red alert. Why? Because passed experience with this company warned me that they were grossly underestimating the task and ignoring firewall permissions, server builds and authorities, and bandwidth. My tech team where equally blasé, relying on the fact that they had stipulated that they need the files by X date and that it was other peoples job to deliver them. I didn't share this sanguine attitude. I started badgering everyone in the whole transmission chain to give me the details of who was doing what, and when it would be done. Needless to say they all had other more pressing problems and without my push we'd never had made the date. Even so it was touch and go.

Interfaces between projects and programs are always a problem. You have to identify them, even if they are not obvious, then interlock with their managers, and constantly manage that interlock. People can forget their commitments: don't let them.

      

It's all about priorities, stupid!

Over previous posts I've bemoaned the incapacity of executives to prioritize projects. Too often like small children in the proverbial candy store they want it all. They are great at willing the ends, but lousy at providing the means. When tasked about this the usual response is to trot out glib sayings: be creative, think outside the box, work smarter not harder. All of which carry the sub-text of: "don't bother me with details that's what I hired you for."

Recently a project I'm working on was hit a by a resources conflict with another pressing project. In all truth the other project had a more compelling case in the short term. What was required was a realization that both could not be delivered has scheduled and that my project's schedule needed to be re-planned. Simple really.

“Not so fast my logical project manager,” was the reply. “We need your project to finish on time. Just tell us what you need to manage this problem and we'll make it happen. Give us the information and we'll escalate with senior management.” Sounded promising.

Well five weeks passed and nothing has happened. We are now at a point where action will be futile. The damage is done and can't be undone. The schedule has slipped but in an uncontrolled manner, with lots of confusion interspersed with bouts of frustration. We can manage most things, but stupidity can only be endured.

Experience of working has a team matters more than the sum of the parts.

The longest battle of the Second World War was the Battle of the Atlantic. It lasted from September 1939 until May 1945, thousands of seamen died and millions of tons of shipping was sunk. The combat between the allied convoys and the U-boats ebbed and flowed, with first one side on top and then the other. Technology and tactics evolved at a startling rate, British innovation and American manufacturing capacity ultimately triumphed, but not without many travails and setbacks.

One of the key findings that the navies discovered was that if a convoy escort group - usually four to six warships - had worked together previously the lower the loss rate of merchantmen from the convoy. Even escort groups comprising ships from different navies performed significantly better than those that had ships from the same navy. It was the experience of working together that was important not the common tradition or language. 

This was such a key factor that the British Admiralty insisted that all escort groups undertake a realistic three day training exercise before every convoy sailing from British ports, even if they had worked together before.

So what does this mean from a project viewpoint? Well it confirms what last blog stressed; that teams are better than ad hoc groupings cobbled together under the matrix management rubric. That time spent by the team familiarizing themselves with the task and how it will be addressed before the project starts is time well spent. A lot of projects do have a Kick Off event, these are only valuable if they address the launch of the project and who's doing what etc. However some are just glorified management ego trips - 'look at how smart we are to have got this project approved. Now don't you workers go and screw it up, we'll be watching!' is their tone.

So get your project's shake down cruise - naval term - done as soon as possible and if you can't have the same team always try to have a comprehensive launch event and activities. Remember Briers law: it is never to soon to start failing!

Team Building and Matrix Management

There are four phases in team building: Forming (team is created), Storming (team debates and argues about what it needs to do), Norming (team agrees how it will work), and Performing (team starts to deliver). The first three phases are time consuming and during this period the team is rarely performing.

So intelligent managers try to ensure that they minimize the amount of team creation they have to undertake. Establishing permanent teams is the most effective from a performance point of view. Replacements or additions to the team are easily incorporated.

Compare this with the situation in a matrix management system. Here the teams are loosely integrated, they are temporary, and they go through the Forming, Storming, and Norming phases on a regular basis - with the resultant negative impact on efficiency and effectiveness.

A clear indicator of project performance is the amount of matrix management involved, the more the matrix the less effectiveness, at least during the all-important starting phase of the project. Matrix management helps to invoke Briers law, which states: that it's never to soon to start failing. The time lost in perpetually re-creating teams greatly affects the ability of teams to deliver projects on time.

Prioritization confusion: A fractal sign

One of the first sign that management has lost the plot is when they are incapable of prioritizing projects.

Once when working as a consultant I had an engagement with a manufacturing company in Bolton, UK. During a tour of the factory I was shown the production controller's office. In there I saw the production schedule board, on that board where displayed all the current work orders with their individual priorities. I noticed that the majority of the orders had a red tag against them, and a large number had two red tags.

I asked the obvious question, "What do the red tags mean?" And I was told that it meant they had the highest priority.

"And the double tags?"

"That's because we had so many red tagged orders that we had to introduce a higher priority, an ultra priority, as it were."

So as a consultant I could see where I could make an easy win. Further as I continued my investigations I discovered that the inability to prioritize the work schedule was systematic of the management team's chaotic style. It wasn't an aberration it was a standard practice. It was a classic example of management fractals: Chaos at the top resulting in chaos at the bottom.

Now in case you think this is a problem confined to manufacturing let me tell you about an episode just before the holidays. I'm in a software management meeting when two department heads said that their different projects were the company's number one priority and that they had first call on key resources. Both were shocked to discover that the other project had a same priority, me, not so much. The senior management has a record of defining multiple number ones in all its activities. Their indecision flows down through the organization. They exhibit fractal management symptoms and, like all such managements, they don't realize it. Prioritization for them is a word, not a practice.