The Exterminators
By: Philip E. Ross
A small British firm shows that software bugs aren't inevitable
Peter Amey was an aeronautical engineer serving in the United Kingdom's Royal Air Force in the early 1980s when he found a serious flaw in an aircraft missile-control system being deployed at the time. It wasn't a defect in any of the thousands of mechanical and electronic parts that constituted the system's hardware. The problem was in the system's software. Amey found an erroneous piece of program code—a bug. Because of it, the unthinkable could happen: under rare circumstances, a missile could fire without anyone's having commanded it to do so.
Amey says his superiors, rather than commending his discovery, complained that it would delay the system's deployment. Like most project managers, they didn't like the idea of fixing errors at the end of the development process. After all, good design ought to keep errors out in the first place. Yet time and again, Amey knew, the software development process didn't prevent bugs; it merely put off dealing with them until the end. Did it have to be that way? Or could developers avoid bugs in the first place? He would find the answer to be "yes" when, years later, he joined Praxis High Integrity Systems.
Praxis, headquartered in Bath, 2 hours from London by car, was founded in 1983 by a group of software experts who firmly believed they could put together a sound methodology to ruthlessly exterminate bugs during all stages of a software project.
At the time, the software world was in a malaise that it hasn't fully shaken even today [see "/sep05/1203" Why Software Fails," in this issue]. Software projects were getting larger and more complex, and as many as 70 percent of them, by some estimates, were running into trouble: going over budget, missing deadlines, or collapsing completely. Even projects considered successful were sometimes delivering software without all the features that had been promised or with too many errors—errors that, as in the missile-firing system, were sometimes extremely serious. The personal computer era, then just starting, only reinforced a development routine of "compile first, debug later."
Praxis armed itself not only with an arsenal of the latest software engineering methods but also with something a little more unusual in the field: mathematical logic. The company is one of the foremost software houses to use mathematically based techniques, known as formal methods, to develop software.
Basically, formal methods require that programmers begin their work not by writing code but rather by stringing together special symbols that represent the program's logic. Like a mathematical theorem, these symbol strings can be checked to verify that they form logically correct statements. Once the programmer has checked that the program doesn't have logical flaws, it's a relatively simple matter to convert those symbols into programming code. It's a way to eliminate bugs even before you start writing the actual program.
Praxis doesn't claim it can make bug-free software, says Amey, now the company's chief technical officer. But he says the methodology pays off. Bugs are notoriously hard to count, and estimates of how common they are vary hugely. With an average of less than one error in every 10 000 lines of delivered code, however, Praxis claims a bug rate that is at least 50—and possibly as much as 1000—times better than the industry standard.
Praxis is still a small, lonely asteroid compared to the Jupiter-size companies that dominate the software universe—companies like Microsoft, Oracle, and SAP. The tiny British software house doesn't make products for the masses; it focuses on complex, custom systems that need to be highly reliable. Such mission-critical systems are used to control military systems, industrial processes, and financial applications, among other things.
Sometimes the software needs to work 99.999 percent of the time, like an air-traffic control program Praxis delivered some years ago. Sometimes it needs to be really, really secure, like the one Praxis recently developed for the National Security Agency, the supersecret U.S. signals intelligence and cryptographic agency, in Fort Meade, Md.
And though Praxis employs just 100 people, its name has become surprisingly well known. "They're very, very talented, with a very different approach," says John C. Knight, a professor of computer science at the University of Virginia and the editor in chief of IEEE Transactions on Software Engineering. Praxis's founders, he says, believed that building software wasn't as hard as people made it out to be. "They thought, it isn't rocket science, just very careful engineering."
Watts S. Humphrey, who once ran software development at IBM and is now a fellow at the Software Engineering Institute at Carnegie Mellon University, in Pittsburgh, also speaks highly of Praxis. He says the company's methodology incorporates things like quality control that should be more widely used in the field. In fact, Humphrey spent this past summer at Praxis headquarters to learn how they do things. He wants to use that knowledge to improve a complementary methodology he developed to help organizations better manage their software projects.
Praxis's approach, however, isn't perfect and isn't for everybody. Formal methods obviously are no silver bullet. For one thing, using formal methods can take more time and require new skills, all of which can mean higher up-front costs for a client. In fact, Praxis charges more—50 percent more in some cases—than the standard daily rate. To this its engineers will say: "You get what you pay for; our bug rate speaks for itself."
And although formal methods have been used to great effect in small and medium-size projects, no one has yet managed to apply them to large ones. There's some reason to think no one ever will, except perhaps in a limited fashion. Nevertheless, even though the company may not have all the answers to make software projects more successful, those working in the field can learn plenty of lessons from it, say advocates like Knight and Humphrey.
A Search is a terrible thing to waste! One Test is Worth 1000 Expert Opinions - The plural of anecdote is not data.
We can't solve problems by using the same kind of thinking we used when we created them. - Unknown
Correctness by Construction: A Manifesto for High-Integrity Software
Martin Croxford, Praxis High Integrity Systems
Dr. Roderick Chapman, Praxis High Integrity Systems
High-integrity software systems are often so large that conventional development processes cannot get anywhere near achieving tolerable defect rates. This article presents an approach that has delivered software with very low defect rates cost-effectively. We describe the technical details of the approach and the results achieved, and discuss how to overcome barriers to adopting such best practice approaches. We conclude by observing that where such approaches are compatible and can be deployed in combination, we have the opportunity to realize the extremely low defect rates needed for high integrity software composed of many million lines of code.
The National Institute of Standards and Technology (NIST) reported in 2002 that low quality software costs the U.S. economy $60 billion per year [1]. According to the aptly named “Chaos Report,” only one quarter of software projects are judged a success [2]. Software defects are accepted as inevitable by both the software industry and the long-suffering user community. In any other engineering discipline, this defect rate would be unacceptable. But when safety and security are at stake, the extent of current software vulnerability is unsustainable.
Recent research on this issue has been conducted on behalf of the National Cyber Security Partnership, formed in 2003 in response to the White House National Strategy to Secure Cyberspace [3]. The partnership’s Secure Software Task Force report states the following:
Software security vulnerabilities are often caused by defective specification, design, and implementation. Unfortunately today, common development practices can often leave numerous defects and resulting vulnerabilities in the complex artifact that is delivered software. To have a secure U.S. cyber infrastructure, the supporting software must contain few, if any, vulnerabilities. [4]
The report goes on to recommend adoption of software development processes that can measurably reduce software specification, design, and implementation defects. It identifies three software engineering practices as examples that satisfy this recommendation. This article describes one of these examples, Correctness by Construction (CbyC), which originates from Praxis High Integrity Systems.
Maturity of Approach
The CbyC approach has two primary goals: to deliver software with defect rates an order of magnitude lower than current best commercial practices in a cost-effective manner, and to deliver durable software that is resilient to change throughout its life cycle.
Elements of the CbyC approach have been used for more than 15 years to produce software with very low defects mainly for safety-critical applications, but more recently for security-critical applications. The approach has evolved over time and now applies to the entire systems development life cycle, from validation of the concepts of operation to preserving correctness properties during long-term maintenance.
CbyC has delivered software with defect rates of less than 0.1 defects/1,000 source lines of code (SLOC) with good productivity: up to around 30 LOC per day. The achieved defect rates compare very favorably with defect rates reported by Capability Maturity Model® Level 5 organizations of 1 defect/1,000 LOC [5]. The comparative rates are shown in Figure 1. It is, of course, true that other approaches have also succeeded in delivering similarly low defect rates, however, it is rare to also deliver good productivity (since low defect rates are often the result of extensive, expensive debugging and testing).
There is more to the article - This is the intro.
The entire article, including diagrams, etc., is attached.
__________________
A Search is a terrible thing to waste! One Test is Worth 1000 Expert Opinions - The plural of anecdote is not data.
We can't solve problems by using the same kind of thinking we used when we created them. - Unknown
Software bugs aren't inevitable - CMM vs CbyC
Linear Mode
