For many years, organizations, researchers, and practitioners have analyzed how to manage technology projects successfully.
Among them is the Standish Group, which regularly publishes its findings in its Chaos reports. In 1994, Standish reported a shocking project success rate of only 16 percent; another 53 percent of the projects were challenged, and 31 percent failed outright. In subsequent reports, Standish updated its findings, yet the numbers remained troublesome.
The numbers indicate large problems with technology projects and have had an enormous impact on software development and project management.
They suggest that the many efforts and best practices put forward to improve how companies manage and deliver technology projects are hardly successful.
Scientific articles and media reports widely cited these numbers. Many authors used the numbers to show that technology project management is in a crisis. The numbers even found their way to a report for the president of the United States to substantiate the claim that U.S. software products and processes are inadequate.
The numbers’ impact and their widespread use indicate that thousands of authors have accepted the Standish findings. They’re perceived as “truth” and unquestionable.
However, the Standish definitions of successful and challenged projects are problematic.
They defined software project success and failure by creating three project categories:
> Resolution Type 1, or project success. The project is completed on time and on budget, offering all features and functions as initially specified.
> Resolution Type 2, or project challenged. The project is completed and operational but over budget and over the time estimate, and offers fewer features and functions than originally specified.
> Resolution Type 3, or project impaired. The project is canceled at some point during the development cycle.
Standish defines a successful project solely by adherence to an initial forecast of cost, time, and functionality. The latter is defined only by the number of features and functions, not the functionality itself.
Standish states the following in their report: “For challenged projects, more than a quarter were completed with only 25 percent to 49 percent of originally specified features and functions.”
This means a project that’s within budget and time but that has less functionality doesn’t fit any category, but I assume that Standish has put these projects that don't comply with one or more of the success criteria into the challenged-project category.
So, Standish defines a project as a success based on how well it did with respect to its original estimates of the amount of cost, time, and functionality.
But in reality, the part of a project’s success that’s related to estimation deviation is highly context-dependent. In some contexts, 30 percent estimation error does no harm and doesn’t impact what the organization would consider project success.
In other contexts, only 5 percent overrun would cause much harm and make the project challenged. In that sense, there’s no way around including more context (or totally different definitions) when assessing successful and challenged projects.
The above illustrates some problems with the definitions. They’re misleading because they’re solely based on estimation accuracy of cost, time, and functionality.
The Standish definitions don’t consider a project’s context, such as benefits, value, and user satisfaction.
Starting with the Chaos Report in 2015, the Standish Group seem to have discovered their own mistake. They changed how they define success. This new method coded the dataset with six individual attributes of success: OnTime, OnBudget, OnTarget, OnGoal, Value, and Satisfaction.
The new definition of successful projects is OnTime, OnBudget, and with a satisfactory result. This means the project was delivered within a reasonable estimated time, stayed within budget, and delivered customer and user satisfaction regardless of the original scope.
This definition encompasses both a success rate for the project management of a project and for the project itself. In my opinion, this improves the definition, since we probably all have seen projects that meet the triple constraints of OnTime, OnBudget, and OnTarget, but the customer was not satisfied with the outcome.
In changing from the OnTarget constraint to Satisfaction, they avoid penalizing a project for having an evolving target, which all projects have, even the very small ones. Customers have a clear opinion on the satisfaction level, whether or not all the features and functions they asked for at the beginning of the project are realized.
They support these changes with their own data. They found that both satisfaction and value are greater when the features and functions delivered are much less than originally specified and only meet obvious needs. And they found that most features and functions of software are not used. These additional features increase cost, risk, and quality but do not necessarily provide value.
But in my opinion, these definitions still have a serious flaw. The Chaos Report, and numerous articles citing it, label canceled projects as “failed” and imply that all of them were canceled because of poor project management.
This implication is both false and dangerous.
It is false because, particularly in an era of rapid change, a lot of technology projects are properly started, well managed, and properly terminated before completion because their original assumptions have changed.
It is dangerous because it often leaves project managers with the following thoughts: “It’s becoming clear that continuing this project will waste company resources. I should probably have the project canceled now, but that would make me the manager of a failed project and wreck my career. I’ll be better off if I say nothing, keep the project going, and look for a new project to transfer to.” See “Why Killing Projects Is so Hard (And How to Do It Anyway).”
So what is project success?
Simply put, project success occurs when the outcome of the project adds value to the organization. And the value of a project is defined by subtracting all of the costs from all of the benefits the project delivers.
Value = Benefits - Costs
This can be roughly translated to three levels of project success:
1) Project delivery success: Will the project delivery be successful? Essentially, this assesses the classic triangle of scope, time, and budget. These are your costs.
2) Product or service success: This refers to when the product or service is deemed successful (e.g., the system is used by all users in scope, up-time is 99.99 percent, customer satisfaction has increased by 25 percent, and operational costs have decreased by 15 percent). These are your benefits.
3) Business success: This has to do with whether the product or service brings value to the overall organization, and how it contributes financially and/or strategically to the business’s success. This is your value.
Overall, a successful project depends on the combination of these criteria. Some argue that product/service success is the same as business success, or argue that product/service success automatically means business success. But this is not true. See “Product or Service Success Does Not Automatically Mean Business Success” for more on this topic.
When you look at project financials and translate the above to dollars, you can fairly say that a project is a failure if it has a negative value. This means the total of all the benefits is lower than the total costs. See “What Are the Real Costs of Your Technology Project?” for more on this topic.
You can even argue that a project is a failure if the targeted return on investment (ROI) is not achieved. This is because you could have done another project with a higher ROI in the same time with the used resources. See “What Are the Real Opportunity Costs of Your Project?” for more on this topic.
But what you should not forget is that project success and project failure are NOT absolutes. It may not be possible to be a little bit pregnant, but you can be a little bit successful.
Every project has multiple success criteria related to business results, product/service results, and project delivery results (cost, schedule, scope, and quality).
Some criteria are absolute, meaning they must be completed on or before the original planned date, and some are relative, meaning they must be completed by a date acceptable to the client.
Project success is determined by how many of your success criteria are satisfied, and how well.
Whether or not a project is successful also depends on who you ask—
> the very happy project manager that implemented the SAP project as scoped on time and below budget (I know, this will NEVER happen),
> the end-users, who absolutely hate the complexity and slowness of the new system,
> or the COO that has seen IT costs double whilst none of the expected savings materialized.
These stakeholders may all have very different opinions on the success of the project.
Project success also depends on when you ask.
Twelve months after the go-live, the users will have a better grasp of the system and initial performance problems will have been solved. And slowly but steadily, the expected savings will often start to materialize as well.
So in order to determine the success or failure of your project, you should define all the criteria relevant to your project, define how you will measure them, and define when you will measure them.
In a nutshell: In order to determine project success (and, as a consequence, project failure), you must define all the criteria relevant to your project, define how you will measure them, and define when you will measure them.
