Can the latest developments in the application development tools market help to close the gap between the developers and the business? Gavin Clarke investigates.
The US Federal Bureau of Investigation (FBI), the government department dedicated to wringing the truth from criminals, had a confession of its own to make in January. Bureau director Robert Mueller testified before a Senate panel that his organisation had invested $170 m and four years developing a case management system, which was still not online. Mueller vowed that he would decide within the following two months whether the system could be salvaged, but put the cost of even scrapping it at $104 m.
The FBI had made three basic mistakes. According to a report by The Justice Department's inspector general, the FBI changed the project's basic goals: shifting from criminal to terrorist investigations in the wake of 11 September. It made a series of poor management decisions early on in the project; and it failed to provide adequate project oversight.
The FBI is not alone in making these errors: businesses worldwide are doing it every day, wasting billions of dollars on software projects that fail to work and which run late.
The impact this has on the business bottom line is huge. According to analysts at The Standish Group, $55 bn is wasted annually on projects in the US alone, while 43% of projects see costs exceed their original budgets. Then there is the unquantifiable cost in unrealised income, lost thanks to new projects not rolling out on time.
The vendors whose tools help organisations to build business applications claim they have an answer to this chaos. Borland Software, IBM Rational and Mercury, along with other ISVs in this sector, are rolling out strategies to ensure application development teams work more effectively and that business managers receive more insight into project status and software performance.
Amnon Landan, Mercury Interactive CEO
Even Microsoft is getting in on the game. Microsoft this year plans to launch an integrated development and collaboration platform, Visual Studio 2005 Team System (VSTS), which it believes will do a better job in helping customers build and manage applications than software offerings from more established rivals.
Prashant Sridharan, senior product manager for the .Net developer product management group, says the status quo has failed customers. "Some vendor had to step up and deliver the lifecycle tools ... this was a market we had to be in because of the scope of the applications people are building," he says.
So why the change? Application lifecycle management (ALM) providers have simultaneously reached a consensus. They believe the application development process is failing. Since 2002, big players like Windows and Java leaders Borland and IBM Rational have been talking less about delivering application development tools and more about integrated 'platforms'.
One reason for this is the tools market has become saturated and commoditised. Free and open source tools proliferate, making it difficult to sell on price alone; while the Eclipse Foundation, founded by IBM, is helping drive commoditisation.
Founded in 2002, Eclipse's original mission was to create an open source framework for Java and C/C++tools. Eclipse has put the work of building IDE menus, bug tracking and common repositories into the community, a move that has provided an open framework for hundreds of plug-in providers to adopt.
The result was that tools providers began casting around for a new message so they could sell to business executives and chief information officers, not just the lowly developers using the free tools. The message they adopted became ALM. In late 2002, Borland bought Together, BoldSoft and Starteam, to enhance its Java and .Net tools with model driven development capabilities, and to provide centralised management of the people and application source code used in projects.
Borland's Java competitor IBM, meanwhile, snapped up ALM heavy-weight Rational Software for $2,1 bn: the second highest figure IBM has ever paid for a software company. Through the deal, IBM not only secured access to a broad application lifecycle management suite, which helped enhance its own Java tools, but it also brought on board some of the leading brains in ALM. These included Grady Booch, co-author of both unified modelling language (UML), used by many to architect their applications, and the rational unified process (RUP), a set of methodologies for teams of people to build applications.
This changing competitive landscape coincided with three industry trends. First, there was a growing body of evidence that application development simply was not working, and that as a result businesses were losing millions of dollars in wasted investment, unachieved total cost of ownership, and unrealised potential income from business projects reliant on software.
The Standish Group's survey of 13 522 IT projects, the Chaos Report, found 82% of projects overran, while just 52% of the required features and functions actually make it into the finished application. Standish Group branded 15% of projects as failures and 51% as 'challenged'.
Both the amount spent and the ROI derived from application development have come under close scrutiny by organisations under tight budget restrictions.
Boz Elloy, senior VP of software products for Borland, says there was a pressing need for vendors to align software development with business. "There was a massive waste in software. Spending was unaccountable, application development was a complete black box. People did not know what the outcome of an application development project would be."
Boz Elloy, Borland senior VP of software products
The next big event to impact application development was the growing taste for outsourcing. Many companies have turned to outsourcers to save money: a priority for 47%, according to management consultant DiamondCluster International.
Employers also believe outsourcers possess the skills they lack to help them turn new technology architectures - Java, .Net, Web services and service oriented architectures - to their companies' advantage. Gartner believes 85% of enterprises will lack the resources and capabilities needed to complete internal application development, with 75% of projects failing partly through a lack of skills.
Sending application development overseas, however, has raised fresh challenges. Teams are now working across time zones and geographies, with engineers attempting to bridge considerable barriers of language and culture.
Software's final challenge is the growing pressure created by a mass of official and government guidelines dictating how the organisation does business. The US Sarbanes Oxley Act (SOX), for example, is not only telling businesses how to act, it is making chief executives personally responsible for ensuring that they comply. Executives must oversee the installation of systems and processes that ensure they are compliant and can be audited to ensure a company's on-going compliance.
Beneath the overall SOX umbrella, vertical sectors face their own regulatory guidelines. Organisations trading in European finance are coping with Basel II, while those in the US health sector must ensure patients' personal information is protected, according to the Health Insurance Portability and Accountability Act of 1996 (HIPAA).
Many have come to believe that regulation can be enforced through business processes, and that software is the best way to capture and enforce these processes. Borland and IBM are increasingly using best practices to specify the sequences of actions developers should take in a project, to ensure ALM flows smoothly and meets the demands of the business and government regulators.
Eric Naiberg, manager of desktop products for IBM Rational, says while companies have established processes in areas like human resources and supply chain management, software itself has been long overlooked. "Organisations are looking at software development like they would a core business process like HR or ERP. They automated those processes and now they are looking at software in the same way."
IBM Rational's answer is to deliver RUP through a set of integrated tools that help individuals model, architect, deliver and manage the software that encapsulates companies' business processes. IBM is partly delivering RUP through a set of integrated Rational and WebSphere tools code-named Atlantic.
Borland, meanwhile, is utilising the capability maturity model (CMM), which forms the basis for the Carnegie Mellon Software Engineering Institute's stringent capability maturity model integration (CMMI). Many organisations, especially outsourcing companies, use CMMI to demonstrate the level of maturity in their software development processes.
Borland moved deeper into CMM by acquiring process automation specialist TeraQuest Metrics in January. TeraQuest had devised a set of templates that encapsulate hard-to-quantify activities in software like change management and project planning. TeraQuest was also home to CMM authors Bill Curtis and Charlie Webber. Borland now plans to encapsulate CMM into its ALM tools, such as the recently launched Core software delivery platform (SDP).
Core SDP is one step in Borland's software delivery optimisation strategy (SDO), launched last year. SDO's goal is to provide a set of role-based environments that deliver just the features required by individual ALM team members, instead of providing individuals with the full suite. These environments are then integrated, to provide improved communications and collaboration across ALM teams.
A vital element in understanding ALM is the digital dashboard. Borland, IBM Rational and Mercury are now beginning to deliver features that combine elements of project management with business analytics, delivered through a set of graphics-based tools to help managers understand the state of software projects.
The digital dashboard is designed to let the business manager assess the likely consequences on a current software development project of adding an extra feature: an action called 'scope creep' that generally introduces delay, as the FBI discovered.
Application performance vendor, Mercury, bought governance specialist Kintana for $225 m in 2003, to help solve the issue of IT governance. Mercury's purchase followed the launch in 2002 of business technology optimisation (BTO), a strategy from Mercury whose phraseology indicated the company's belief in bringing the management and performance of software inline with business needs.
Mercury's chief marketing officer, Christopher Lochhead, says the company's IT Governance Centre, from Kintana, helps customers avoid scope creep. "Scope creep kills software projects. You are building an application with 10 critical function points and a month into the design line the business wants 15. That extends the project by 10 to 15 months and IT gets the blame."
The convergence between business and application development is a trend that Microsoft has now picked up on. For nearly 30 years Microsoft stayed on ALM's sidelines, preferring to partner with Borland and Rational, among others. These companies provided tools in the areas of requirements gathering, application modelling and team-based development that helped extend Microsoft's Visual Studio's relatively basic capabilities.
This summer, however, Microsoft will launch the VSTS platform. VSTS includes an environment, code-named Whitehorse, which enables ALM team members to model and deploy applications, in addition to a centralised server, Team Server, that will run on Microsoft's SQL Server and provide an infrastructure to help manage projects.
The important thing for business managers, according to Microsoft's Sridharan, is VSTS's integration with Microsoft's existing line-of-business applications such as Office's Project, Excel and Outlook, due in version 1.0 of the product. Sridharan says this is important because it enables business managers to tell application development team members what their project needs are early on in a software project's lifecycle, using a set of tools that managers are already familiar with.
Despite its relative newness to ALM, Microsoft believes it can win business by offering a combination of easy-to-use tools, many of which are already familiar to customers, at a low price.
Despite its desire to stand out on price and features, Microsoft is actually following Borland, Rational and Mercury, by delivering a vision for systems development and management that recognises the increasing complexity of today's distributed software landscape. These architectures are founded on a growing mix of .Net, Web services, SOAs, Java, Windows, Perl, Python, Cobol and mainframe technology.
Against this background, Borland's Elloy says it is more important than ever for businesses to get a grip on their software infrastructure. Elloy estimates the average lifespan of an application is eight years, during which time it will have to be upgraded, integrated with other systems, bugs will need fixing, and the application will probably need to be scaled out to an increased number of users.
To Elloy, the answer is process and process means bringing in business: "The business guy knows about process ... this is about process, from requirement to requirement [of the application]."
Years ago, application development and business operated in silos. Years of fiscal frugality combined with new, quickly changing business objectives, have forced IT to become more accountable for its actions. And, increasingly, ALM providers are delivering the kinds of tools that not only help the business quantify what the developers are doing, but which also help ALM teams become more effective.
Software development has, for too long, been more art than science, plagued by overspending and poor delivery problems. While vendors are now headed in the right direction with changes in products and strategies to improve the development process and involve more business input, CBR recommends caution. ISV's product strategies are in different stages of evolution, meaning customers should carefully scrutinise different vendors' capabilities before purchase.