Including Risk Analysis in Software Cost Estimates

In spite of the fact that their outputs may not always be believed, modern software cost estimating tools are now capable of serving a variety of important project management functions.

by Capers Jones, President, Capers Jones & Associates LLC

Software has a bad reputation as a troubling technology. Large software projects have a very high frequency of schedule overruns, cost overruns, quality problems, and outright cancellations. If the software is developed under contract by an outsource vendor, many of these problems end up in court for breach of contract.

From studies carried out by the author and his colleagues, manual estimates by project managers tend to be more optimistic than automated estimates carried out by commercial estimating tools such as COCOMO II, CostExpert, KnowledgePlan, Price-S, SEER, SLIM, and a number of others. The reason for this is that commercial software estimating tools are pre-loaded with substantial volumes of historical data. Commercial estimating tools don’t “forget” to include activities such as user documents. They also use historical data for dealing with critical activities such as test schedules.

In some lawsuits, depositions and testimony reveal the surprising fact that accurate estimates were prepared using automated tools, but the estimates were rejected by top executives or clients. In place of the original estimates, the project managers were ordered to meet arbitrary schedules and costs established by clients based on external business needs rather than team capabilities.

This situation brings up one of the first risks of software applications: accurate estimates may not be believed. The best solution to this problem is to support the estimate by large volumes of historical benchmark data.

In spite of the fact that their outputs may not always be believed, modern software cost estimating tools are now capable of serving a variety of important project management functions. In addition, the accuracy and precision of such tools is now high enough to merit their use for business agreements such as contracts and outsource agreements.

Three Fundamental Equations of Software Estimating

Although software cost estimating is a very difficult intellectual problem, there are three fundamental equations that are rather straightforward. These equations make use of important estimating concepts that are linked together for estimation:
1) assignment scopes, 2) production rates, 3) schedules.

The “assignment scope” is the amount of some kind of work for which one person will normally be responsible. Assignment scopes can be expressed in terms of natural metrics such as pages of specifications or lines of code. However assignment scopes can also be expressed in terms of synthetic metrics such as function points. The use of function point metrics allows all activities to be estimated using a single metric.

The “production rate” is the amount of work which one person can complete in a given time period such as an hour, a day, a week, a month, or a year. Here too either natural or synthetic metrics can be used.

The schedule of an activity or task is the amount of calendar time required to complete an activity. Schedules can be calculated by dividing the effort required for an activity by the staff available to perform it. An example can clarify the three concepts.

Suppose you are managing an application of 100,000 lines of source code using the C programming language. A typical assignment scope for an average C programmer is roughly 10,000 lines of source code, so this application might require 10 programmers.

Journeyman C programmers have average coding production rates of about 2,000 lines of source code per month, so the coding of the application will take about 50 person-months of effort.

If the effort for coding the project is 50 person-months and there are 10 programmers assigned, then the schedule for coding will be 5 calendar months.

Note that the logic of assignment scopes and production rates can be expressed in almost any metric: lines of code, function points, pages of text, test cases, or whatever. The normal sequence of applying these estimating equations is:

1. Size of deliverable / assignment scope = staff

2. Size of deliverable / production rate = effort

3. Effort / staff = schedule

The equations are simple in principle, but very tricky in real life. The difficulty is not in the equations themselves, but in knowing the exact assignment scopes and production rates with enough precision for the equations to yield useful results. It is the ability to adjust the fundamental equations in response to varying degrees of staff skill, to different tools, to different methodologies, and to different programming languages that make commercial software estimating tools valuable. This same kind of information is so difficult to acquire and so valuable that the estimating tool vendors almost universally treat adjustment data as proprietary information or as trade secrets.

Nine Steps of Software Cost Estimation

All software estimating tools typically perform the eight of the nine functions discussed here: 1) Sizing project deliverables, 2) Estimating defect levels and removal efficiency; 3) Selecting project activities; 4) Estimating staffing levels; 5) Estimating effort; 6) Estimating costs; 7) Estimating schedules; 8) Estimating requirements changes during development.

The ninth function is not always present in software cost estimation tools: risk analysis. The major risks that need to be analyzed include:

1. Outright cancellation of the project

2. The odds of litigation for breach of contract

3. Poor quality control

4. Excessive requirements changes

We will discuss risk estimation after we have examined the eight steps in the normal sequence of estimation, although some risks are included in normal estimating steps.

Step 1: Sizing Specifications, Source Code, and
Test Cases

The first step in any software estimate is to predict the sizes of the deliverables that must be constructed. Sizing must include all deliverable such as specifications, documents, and test cases as well as source code. The older software cost estimating tools such as the original COCOMO did not include sizing logic, and size information had to be provided by the user.

However, the invention of function point metrics has made full sizing logic for all deliverables a standard feature of commercial estimating tools. As of 2008, sizing is a standard feature of commercial software cost estimating tools, and a variety of sizing methods are now included, such as:

1. Sizing based on function point metrics

2. Sizing based on lines of code (LOC) metrics

3. Sizing based on object-oriented metrics

4. Sizing based on analogy with past projects

5. Sizing based on pattern-matching

6. Sizing based on attributes such as screens and reports created

It should be noted that one very common risk with estimates based on “lines of code” metrics is that such estimates are not reliable for predicting user documents or any non-coding activity such as quality assurance, data base administration, and project management. LOC-based estimates and function point-based estimates are of approximately equal accuracy for predicting coding activities, but the LOC estimates usually are less accurate for non-code activities. Since paperwork in all of its forms is often the most expensive task for large defense applications, this problem is fairly significant. Even for civilian projects paperwork may cost more than the source code for large applications.

Step 2: Estimating Defects and Defect Removal Efficiency Levels

A key aspect of software cost estimating is predicting the time and effort that will be needed for design reviews, code inspections, and all forms of testing. In order to estimate defect removal costs and schedules, it is necessary to know about how many defects are likely to be encountered.

Poor quality control is another major risk that can lead to litigation. In every case where the author worked as an expert witness, quality control was deficient. Lack of early defect detection and removal via inspections can lead to huge delays in testing schedules. What happens is that testing might start on time, but due to the unexpected volume of defects it cannot end on time. Testing is the primary phase where schedules begin to go out of control.

The typical sequence is to estimate defect volumes for a project, and then to estimate the series of reviews, inspections, and tests that the project utilize. The defect removal efficiency of each step will also be estimated. The effort and costs for preparation, execution, and defect repairs associated with each removal activity will also be estimated.

Table 1 illustrates the overall distribution of software errors. In Table 1, bugs or defects are shown from five sources: requirements errors, design errors, coding errors, user documentation errors, and “bad fixes.” A “bad fix” is a secondary defect accidentally injected in a bug repair. In other words, a bad fix is a failed attempt to repair a prior bug that accidentally contains a new bug. On average about 7% of defect repairs will themselves accidentally inject a new defect, although the range is from less than 1% to more than 20% bad fix injections.

Table 1 presents approximate average values, but the range for each defect category is more than 2 to 1. For example, software projects developed by companies who are at level 5 on the SEI Capability Maturity Model (CMM) might have less than half of the potential defects shown in Table 1. Similarly, companies with several years of experience with the “Six Sigma” quality approach will also have lower defect potentials than those shown in Table 3. Several commercial estimating tools make adjustments for such factors.

Note that the majority of defects shown in Table 1 are not in the code itself. This is one of the reasons why LOC-based estimation is less accurate than function-point based estimation. Estimates based on lines of code cannot accurately predict bugs or defects in requirements and design, which outnumber coding defects for all large applications.

A key factor for accurate estimation involves the removal of defects via reviews, inspections, and testing. The measurement of defect removal is actually fairly straight forward, and many companies now do this. The U.S. average is about 85%, but leading companies can average more than 95% removal efficiency levels.

It is much easier to estimate software projects that use sophisticated quality control and have high levels of defect removal in the 95% range. This is because there usually are no disasters occurring late in development when unexpected defects are discovered. Thus, projects performed by companies at the higher CMM levels or by companies with extensive six-sigma experience for software often have much greater precision than average.

Step 3: Selecting Project Activities

Once the size of various deliverables has been approximated the next step is to determine which specific activities will be carried out for the project being estimated. This is one of the major areas where software cost estimating tools excel. Modern cost estimating tools can analyze the nature, size, and class of the application being estimated and automatically select the most likely set of activities.

Estimates for entire projects or for phases are not accurate enough for contracts or serious planning. Activity-based cost estimates with perhaps 20 to 25 activities are the level of precision offered by modern cost estimating tools.

Step 4: Estimating Staffing Levels

Although staffing, effort, costs, and schedules are all important for the final estimate, the normal place to start estimating is with staffing levels. The fundamental equation for determining staff is:

Size of deliverable / assignment scope = staff.

Commercial software cost estimating tools apply this fundamental staffing equation in a wide variety of forms, including but not limited to:

Pages of specifications / assignment scope = analysts

Lines of source code / assignment scope = programmers

Test cases / assignment scope = testers

Pages of user manuals / assignment scope = technical writers

Number of employees / assignment scope = managers

The examples just shown make use of natural metrics that define the actual material to be created such as pages or code. The staffing equation can also be used with the synthetic function point metric. In fact, the equations can be utilized with any metric if it can express the amount of work normally assigned to individual technical workers.

Step 5: Estimating Software Effort

The term “effort” defines the amount of human work associated with a project. The amount of effort can be expressed in any desired metric, such as work hours, work days, work weeks, work months, or work years. Usually small projects of up to perhaps 1000 function points utilize “hours” for expressing effort, but the larger projects in excess of 10,000 function points normally utilize “months” as the unit of measure. The general algorithm for predicting effort is:

Size of deliverable / production rate = staff effort

Here too this basic equation is used in a variety of forms including but not limited to:

Pages of specifications / production rate = analyst months

Lines of source code / production rate = programmer months

Test cases / production rate = testing months

Defects found / production rate = rework months

Pages of user manuals / production rate = writing months

Note also that the use of “months” in this example is simply to indicate a work period. Estimates can be expressed in terms of work hours, work weeks, work months, or even work years.

Some typical examples of these rules are as follows: Specifications are normally prepared at a rate of about 20 pages per analyst month while technical writing proceeds at a rate of about 50 pages per month. Coding takes place at a rate of about 1000 source code statements per month. Here, too, there are very broad ranges and so average values need to be adjusted to match the actual project and team characteristics.

For any given software activity, the measured range between the best performance and the worst performance is about 1000%. This is far too broad a range for “average” values to be useful for serious estimation. Therefore, knowledge of how to adjust production rates in response to various factors is the true heart of software estimation.

Step 6: Estimating Software Costs

The fundamental equation for estimating the cost of a software activity is simple in concept, but very tricky in real life:

Effort * (salary + burden) = cost

One basic problem is that software staff compensation levels vary by about a ratio of 3 to 1 in the United States, and by more than 10 to 1 when considering global compensation levels for any given job category. For example, here in the United States there are significant ranges in average compensation by industry and also by geographic region. Programmers in a large bank in mid-town Manhattan or San Francisco will average more than $80,000 per year, but programmers in a retail store environment in the rural south might average less than $55,000 per year.

With cost ranges such as those for projects that took exactly the same amount of programming staff effort, it is easy to see why figures such as “average cost per function point” should not be used without substantial adjustments for local conditions.

Step 7: Estimating Software Schedules

The fundamental equation for estimating the schedule of any given software development activity is:

Effort / staff = time period

Using this equation, an activity that requires eight person-months of effort and has four people assigned to it can be finished in two calendar months; i.e. 8 months / 4 people = 2 calendar months.

However, in real life, schedule estimating is one of the most difficult parts of the software estimation process and many highly complex topics must be dealt with, such as:

• Dependencies of one activity upon previous activities

• Overlap or concurrency of activities

• The critical path through the sequence of activities

• Less than full-time availability of staff

• Number of shifts worked per day

• Number of effective work hours per shift

• Paid or unpaid overtime applied to the activity

• Interruptions such as travel, meetings, training, or illness

Incidentally, there are two main reasons for schedule slippages on large software applications: 1) Excessive defect volumes that do not appear until testing begins or even in the midst of testing; 2) Significant volumes of requirements changes that were not planned for or included in the schedules.

Step 8: Estimating Requirements Changes During Development

One important aspect of estimating is dealing with the rate at which requirements “creep” and hence make projects grow larger during development. Fortunately, function point metrics allow direct measurement of the rate at which this phenomenon occurs, since both the original requirements and changed requirements will have function point counts.

Changing requirements can occur at any time, but the data in Table 2 runs from the end of the requirements phase to the beginning of the coding phase. This time period usually reflects about half of the total development schedule. Table 2, shows the approximate monthly rate of creeping requirements for six kinds of software, and the total volume of change that might be anticipated:

For estimates made early in the life cycle, several estimating tools can predict the probable growth in unplanned functions over the remainder of the development cycle. This knowledge can then be used to refine the estimate, and to adjust the final costs in response.

Of course, the best response to an estimate with a significant volume of projected requirements change is to improve the requirements gathering and analysis methods. Thus projects that use prototypes, joint application design (JAD), requirements inspections, and other sophisticated requirements methods can reduce later changes down to a small fraction of the values shown in Table 2. Indeed, the initial estimates made for projects using JAD will predict reduced volumes of changing requirements.

Step 9: Software Risk Analysis

The software industry has long been troubled by major schedule slippage, major cost overruns, and a high incidence of outright failure.

Table 3 shows the approximate frequency of various kinds of outcomes, based on the overall size of the project being attempted. Table 3 is taken from the author’s book, Patterns of Software Systems Failure and Success (International Thomson Press, 1995).

As can easily be seen from Table 3, small software projects are successful in the majority of instances, but the risks and hazards of cancellation or major delays rise quite rapidly as the overall application size goes up. Indeed, the development of large applications in excess of 10,000 function points is one of the most hazardous and risky business undertakings of the modern world.

Of all the troublesome factors associated with software, schedule slips stand out as being the most frequent source of litigation between outsource vendors and their clients. Schedule slips are also the main reason for executive frustration with software for internal projects.

Fortunately, as of 2008, objective empirical data is beginning to become available in significant quantities. The International Software Benchmark Standards Group (ISBSG) was founded in 1997. Now that it has been operational for more than 10 years, the volume of measured historical data has reached about 5,000 software projects.

Besides ISBSG there are a number of software benchmark data bases, but these are usually proprietary and not generally available. Some of this data is generally available in books such as the author’s Applied Software Measurement (McGraw Hill, 2008), Software Assessments, Benchmarks, and Best Practices (Addison Wesley 2000), and Estimating Software Costs (McGraw Hill 2007).

Estimating the Risk and Costs of Canceling Large Software Applications

For applications larger than 10,000 function points in size, cancellation occurs with alarming frequency as shown in Table 4. The most troubling aspect of cancelled projects is the fact that the projects are usually late and over budget at the time of cancellation. On average, cancelled projects cost about 15% more than successful projects of the same size.

Since successful applications in the 10,000 function point size range cost about $2,000 per function point, it is a serious waste of corporate funds that cancelled projects average about $2,300 per function point. Larger applications cost even more.

Estimating the Risk and Costs of Litigation

When internal projects are cancelled, money is wasted and sometimes people are fired, but the situation does not end up in court. When projects done by outsource vendors or contractors are cancelled, the situation often does end up in court. Table 4 shows the approximate results of U.S. outsource agreements:

In performing “autopsies” of cancelled or failed projects it is fairly easy to isolate the attributes that distinguish disasters from successes. Experienced project managers know that false optimism in estimates, failure to plan for changing requirements, and inadequate quality approaches lead to failures and disasters. Conversely, accurate estimates, careful change control, and top-notch quality control are stepping stones to success.

If a cancelled project does end up in court, both parties can look forward to about 36 months of expensive litigation, unless they settle out of court. For an application of 10,000 function points in size, the plaintiff can expect to spend at least $5,000,000 in legal fees and expert witness costs. This is about $500 per function point. The defendant will probably need to spend about $7,000,000 in legal fees and expert witness costs, which is about $700 per function point.

Table 4: Approximate Distribution of U.S. Outsource Results
after 24 Months

Since the loser of the case may end up paying the legal fees and costs of both sides, the total legal costs can top $12,000,000 or $1,200 per function point. This does not include any fines, damages, or other awards that judges or juries might award to the winning side.

Not only are there directs costs for legal fees and expert witnesses, but both parties can expect to lose at least 6 months of productive and effective time on the part of the managers and executives who were involved in the project that is under litigation.

At the end of the day, neither the plaintiff nor the defendant is likely to end up ahead. Litigation is usually a lose-lose situation where neither party gains much of value. This brings up the final point of litigation risk analysis. Contracts should be clear and unambiguous about four key topics:

1. Changes to requirements

2. Quality control activities

3. Volumes of delivered defects

4. Progress tracking during development

Litigation and software project failures are an unfortunate byproduct of poor training and preparation on the part of management on both sides of the case.

Summary and Conclusions

Software estimating is simple in concept, but difficult and complex in reality. The difficulty and complexity required for successful estimates exceeds the capabilities of most software project managers to produce effective manual estimates.

The commercial software estimating tools can often outperform human estimates in terms of accuracy, and always in terms of speed and cost effectiveness. However, no method of estimation is totally error-free. The current “best practice” for software cost estimation is to use a combination of software cost estimating tools coupled with software project management tools, under the careful guidance of experienced software project managers and estimating specialists.

In addition to normal development estimation, large projects in the 10,000 function point size range should also include specific risk estimates and deal with the problems that are known to cause trouble: 1) Estimating accuracy; 2) Quality control; 3) Change control; 4) Status reporting.

The strongest point that can be made is that producing excellent software is cheaper and takes less time than producing buggy software that is likely to fail or run late. Producing software that is cancelled or ends up in court will be between 15% and 250% more costly than creating excellent software of the same size and type. To quote Phil Crosby’s famous book, “Quality is Free.” For software, not only is quality free but it costs a great deal less than buggy software and can be produced faster as well.

About The Author

Capers Jones is currently the President and CEO of Capers Jones & Associates LLC. He is also the founder and former chairman of Software Productivity Research LLC (SPR). He holds the title of Chief Scientist Emeritus at SPR. Capers Jones founded SPR in 1984. Before founding SPR Capers was Assistant Director of Programming Technology for the ITT Corporation at the Programming Technology Center in Stratford, Connecticut. He was also a manager and researcher at IBM in California.

Capers is a well-known author and international public speaker. Some of his books have been translated into six languages. All of his books are translated into Japanese and his newest books are available in Chinese editions as well. Among his book titles are Patterns of Software Systems Failure and Success (Prentice Hall 1994), Applied Software Measurement, 3rd edition; (McGraw Hill 2008), Software Quality: Analysis and Guidelines for Success (International Thomson 1997), Estimating Software Costs, 2nd edition (McGraw Hill 2007), and Software Assessments, Benchmarks, and Best Practices (Addison Wesley Longman 2000). The 3rd edition of his book Applied Software Measurement was published in the Spring of 2008.

Author Contact Information

Capers Jones: CJonesIII@cs.com

References and Readings

Charette, Bob; Software Engineering Risk Analysis and Management; McGraw Hill, New York, NY; 1989.

Charette, Bob; Application Strategies for Risk Management; McGraw Hill, New York, NY; 1990.

Crosby, Phil; Quality is Free; Mentor Executive Library, New York, NY; 1979.

DeMarco, Tom; Controlling Software Projects; Yourdon Press, New York; 1982; ISBN 0-917072-32-4; 284 pages.

Ewusi-Mensah, Kweku; Software Development Failures; MIT Press, Cambridge, MA; 2003; ISBN 0-26205072-2276 pages.

Galorath, Dan; Software Sizing, Estimating, and Risk Management: When Performance is Measured Performance Improves; Auerbach Publishing, Philadelphia; 2006; ISBN 10: 0849335930; 576 pages.

Garmus, David and Herron, David; Function Point Analysis – Measurement Practices for Successful Software Projects; Addison Wesley Longman, Boston, MA; 2001; ISBN 0-201-69944-3;363 pages.

Gilb, Tom and Graham, Dorothy; Software Inspections; Addison Wesley, Reading, MA; 1993; ISBN 10: 0201631814.

Glass, R.L.; Software Runaways: Lessons Learned from Massive Software Project Failures; Prentice Hall, Englewood Cliffs; 1998.

Johnson, James et al; The Chaos Report; The Standish Group, West Yarmouth, MA; 2000 (produced annually).

Jones, Capers; Applied Software Measurement; McGraw Hill, 3rd edition 2008; ISBN 978-0-07-150244-3; 575 pages; 3rd edition due in the Spring of 2008.

Jones, Capers; Assessment and Control of Software Risks; Prentice Hall, 1994; ISBN 0-13-741406-4; 711 pages.

Jones, Capers; Patterns of Software System Failure and Success; International Thomson Computer Press, Boston, MA; December 1995; 250 pages; ISBN 1-850-32804-8; 292 pages.

Jones, Capers; Software Quality – Analysis and Guidelines for Success; International Thomson Computer Press, Boston, MA; ISBN 1-85032-876-6; 1997; 492 pages.

Jones, Capers; Estimating Software Costs; McGraw Hill, New York; 2007; ISBN 13-978-0-07-148300-1.

Jones, Capers; Software Assessments, Benchmarks, and Best Practices; Addison Wesley Longman, Boston, MA; ISBN 0-201-48542-7; 2000; 657 pages.

Jones, Capers: “Sizing Up Software;” Scientific American Magazine, Volume 279, No. 6, December 1998; pages 104-111.

Jones, Capers; Conflict and Litigation Between Software Clients and Developers; Software Productivity Research, Inc.; Narragansett, RI; 2008; 45 pages.

Jones, Capers; “Preventing Software Failure: Problems Noted in Breach of Contract Litigation”; Capers Jones & Associates, Narragansett, RI; 2008; 25 pages.

Kan, Stephen H.; Metrics and Models in Software Quality Engineering, 2nd edition; Addison Wesley Longman, Boston, MA; ISBN 0-201-72915-6; 2003; 528 pages.

Radice, Ronald A.; High Qualitiy Low Cost Software Inspections; Paradoxicon Publishingl Andover, MA; ISBN 0-9645913-1-6; 2002; 479 pages.

Wiegers, Karl E.; Peer Reviews in Software – A Practical Guide; Addison Wesley Longman, Boston, MA; ISBN 0-201-73485-0; 2002; 232 pages.

Yourdon, Ed; Death March - The Complete Software Developer’s Guide to Surviving “Mission Impossible” Projects; Prentice Hall PTR, Upper Saddle River, NJ; ISBN 0-13-748310-4; 1997; 218 pages.

Yourdon, Ed; Outsource: Competing in the Global Productivity Race; Prentice Hall PTR, Upper Saddle River, NJ; ISBN 0-13-147571-1; 2005; 251 pages.