With or Without a Navigator? - It’s Your Call

By Terry Wright, Department of Premier and Cabinet, State of Victoria, Australia

As the battered and bruised project manager of several major software engineering projects, by the late 1980s I had had enough. Enough of being expected to deliver to budgets and timelines that were clearly unrealistic. Enough of reporting to project boards that didn’t believe a word I was saying. Enough of the feel-good rhetoric within the software engineering profession about improving productivity and adopting best practice when we had no idea what constituted either.

The cynicism of our customers of the software engineering industry’s ability to deliver was well founded. Around 30% of projects failed to deliver anything and, of those that did complete, the average blow out was in excess of 80%. Business managers were living in fear of failure of their critical software investments.

I had a life-long friend who was much smarter than me. He had decided on a career as a project manager in building construction and, as I often pointed out over a few drinks, his job was much easier than mine. His industry had a large knowledge base of past projects and a specific profession dedicated to interpreting and applying that knowledge so they could predict, manage changes and benchmark productivity. He had a Quantity Surveyor sitting next to him advising him of the differences between fact and fantasy, the likely impact of changes and generally controlling the scope. His situation was similar to that of the long-distance aeroplane pilots of yesteryear who had navigators that told them exactly where to fly to get to their destination while avoiding potential risky situations.

My frustration was being shared globally and resulted in the creation of a number of national not-for-profit industry bodies that were focussed on providing some real data that would improve our ability to manage software engineering projects. The discipline of creating fact-based knowledge to support software engineering was globally known as software metrics.

Creation of a Knowledge Base for Software Engineering Projects

With the growth of these new national software metrics organisations in the late 80s and early 90s, a cooperative global movement started and continues to this day. This resulted in the development of an effective measure of the size of the software engineering product (functional size measure), the definition of ‘productivity’ as it relates to software engineering and the collection of the data of completed projects against these standards. Productivity was defined as the number of hours it takes to deliver one functional unit of software into production. It is expressed as ‘hours per function point’. Specific definition of when a project starts and ends and the effort levels that are included can be found in the data collection questionnaire that can be downloaded at www.isbsg.org. The progress made in this area has been outstanding. In 1990 there was no standard definition of productivity and no publicly available data. Today the International Software Benchmarking Standards Group (ISBSG) has a publicly available repository and learnings based on over 4,000 validated projects from 40 countries. Understanding and applying this knowledge is a specialized skill and not one that all project managers can be expected to have.

By 1995 it became clear that, if the knowledge we had amassed was properly applied, I would be as capable as my friend in building construction in developing early and defendable estimates of size, cost and duration, in managing change and in performance benchmarking.

Time for Change…..The Creation of southernSCOPE

When it came to acquiring custom-built software, government and business had two buying options. The first was ‘fixed price’ where they specify their requirements and assess quotations from suppliers to select the ‘best’ quote. The second was ‘time and materials’ where they pay a preferred supplier (or their internal staff) on the basis of time spent developing the product.

The common approach our industry takes in developing software requirements also seemed to me to be flawed to the point that in most cases it will inevitably result in tears. If I approach an architect to design a new home his first question to me is, ‘What is your budget?’ Based on this budget the remainder of our discussions centre on the features I most need and how these might be delivered within the nominated budget. We don’t do this in software engineering. Our architect starts by helping the customer create a comprehensive ‘dream list’ of anything we can both think of that might be nice. It is an exhilarating experience. Upon completion, we use our individual brands of black magic to arrive at the likely delivery cost (complete with whatever caveats we can think of). It is normal for the cost to be many times what the customer was prepared to pay, and at this point, the honeymoon is well and truly over. The relationship is in demise.

In 1995 with our newfound knowledge in hand, it was time to try to right a few of the old wrongs. The Government of the State of Victoria in Australia developed southernSCOPE, an alternative approach to acquiring custom built software that would try to address some of the imbedded problems. It also established a policy making southernSCOPE the preferred method that agencies of government should use to acquire their custom-build software. In 2000 a study of the extent and impact of its use was undertaken and resulted in the release of a revised approach. Although the number of projects that had used the approach was small their nature was diverse. The results were outstanding:

• All projects completed within 10% of the initial project budget
• They all had a high customer satisfaction in that the software met the intended business need
• Their cost per unit was in the lowest 25% of comparative industry benchmarks.

Why then is the southernSCOPE method so effective? The method successfully enabled the following problems inherent in the software engineering process to be addressed:

• Realistic cost estimates are provided at project inception
• The functional requirements developed and agreed to are sound and unambiguous
• The customer is able to make objective decisions in language he understands as to what functionality should be provided within the agreed budget
• Scope creep is mitigated

How does southernSCOPE work?

In effect, southernSCOPE is just an aggregation of the practical capabilities made possible through the new knowledge base applied at various places in the system development lifecycle. Whereas for the maximum benefit it should be applied from project inception it can also be applied at several points throughout the project lifecycle with good impact. The approach comprises the following steps (see Table 1):

What is the Role of the ‘Scope Surveyor’?

Critical to the above approach is the engagement of a suitably skilled ‘Scope Surveyor’ that is contracted to the project throughout the development lifecycle by the customer/acquirer. This person provides the early budget estimate (Step 1), leads in the development of the Project Requirements document (Step 2), overseas the completeness of the functional requirements document (Step 3) and the development of the Function Baseline (Step 4), monitors the delivery of the required functionality and any change management issues (Steps 5 and 6) and plays a key role in the finalisation of the project (Step 6). They must be expert in the global software metrics knowledge base and its practical application as well as having a strong background in software engineering. The cost of this resource is normally between 1 and 2 percent of the project cost. Since part of their role is to be a ‘referee’ about what functionality is in and out of scope, there is an escalation process to an independent authority for use in the case of a dispute between the supplier and purchaser. To my knowledge, this escalation has never been enacted. As with the building engineering industry, as our industry matures, for particularly large projects both the supplier and the purchaser may elect to engage Scope Surveyors to provide key expert cost, scope and delivery functions to both parties.

The southernSCOPE approach is particularly effective when the functional requirements are not well known as it provides for the analysis and ultimate selection of the functionality that will best meet the business need. It is like having a ‘time-box of functionality’ to which the project works. An early estimate of functional size for an identified business might indicate that, for example, 800 functional units of software would be sufficient to meet that need. Once the quantity is established, the project activity that follows focuses on identifying what specific functionality should fit into the allocation of functional size and then ensuring it is developed and delivered. As functional size measures (FSMs) have now been developed that are particularly suited to embedded real time systems (such as COSMIC) the southernSCOPE should have similar relevance in these environments.

How widespread is its use?

This is the sad part. This approach was developed ten years ago. Five years ago a study of its effectiveness found it to be both innovative and able to solve some long standing problems. Today, its known use is scattered a little around Europe and in Brazil. Within the Victorian government, there has been scattered use of the approach mainly on smaller projects but there have been some interesting developments.

The Victorian Government has an Office of the Chief Information Officer that has the responsibility for driving best practices across the Victorian government. To this end, since 2002 that Office has funded the engagement of a Scope Surveyor for four projects critical to government. In each case, the story has been the same (see Table 2):

Table 2: Observations of Recent Implementations

Although there has been no formal analysis of this pattern of behaviour of project managers, one can speculate on some possible reasons as follows:

• Project managers don’t know how to segment their work to enable the scope surveyor to fit within the team
• They feel threatened and somehow impotent in that someone is doing the job they should do
• They know the old ‘fixed price’ scenario well and are comfortable operating with it
• They react to a pushback from vendors that would much prefer the old fixed price as they also are comfortable with it and it also gives them the flexibility to drive 20 – 80% scope creep in ‘authorised changes’. This method eliminates that.

It seems that thirty years of entrenched habits within what have traditionally been the world’s leading software engineering countries mean that the adoption of new ideas and practices is slow --- very slow. This contrasts with the rapidly evolving new economies that don’t have the same ‘baggage’ and are looking to adopt the current best practice at entry level. A good example of this is South Korea where the government is proactive in driving the best practices that the new software engineering knowledge base has enabled.

The southernSCOPE methodology was developed by the Government of Victoria. The methodology and supporting documentation is freely available at http://www.egov.vic.gov.au/index.php?env=-innews/detail.tpl:m1816-1-1-7:l0-0-1-:n832-0-0

About the Author
Terry Wright is currently a strategist with the Department of Premier and Cabinet of the government of the state of Victoria in Australia. He is the developer of the southernSCOPE methodology and, since 1990, has been a key driver in the creation and exploitation of the current body-of-knowledge of software engineering projects so that the world can better understand and improve the management of software engineering projects. In 1995 he was the founder and, until 2005 the President, of the International Software Benchmarking Standards Group (ISBSG), the organization that has established a repository of over 4,000 projects from more than forty countries. Over the past decade Terry has been at the forefront of innovation in the way that governments use information and communication technologies to transform the way government operates and delivers services.

Author Contact Information
Terry Wright, Strategy and Futures
Office of the Chief Information Officer
Department of Premier and Cabinet
3 Treasury Place, East Melbourne, 3002
Office: 96511182 Mobile: 0412 262 890
Email: Terry.Wright@dpc.vic.gov.au