Blog root page
previous page in category
next post in category
The was a "quality revolution" in the '80s for manufacturing of all kinds. It was led by the Japanese but it was closely followed by the rest of the PacRim. Previously the conventional wisdom was that one could "test in" quality by finding defects and removing them (either fixing the defect or not shipping the part). The quality revolution changed all that by demonstrating that quality is a process issue and testing is inadequate for ensuring correctness or reliability in a product. That was demonstrated in the marketplace when PacRim manufacturers grabbed huge chunks of market share from traditional Western manufacturers simply because their products worked while the others' didn't.
That revolution profoundly changed the way Western manufacturers viewed quality and they had to d a whole lot of scrambling to scrap the old paradigm and install the new one. However, by the '90s the new paradigm was firmly ensconced throughout all manufacturing activities. It even migrated into service industries and pure engineering disciplines.
A relevant anecdotal data point. In '79 I was hired by a company that made Big Honking Automated Test Equipment (ATE) for the electronics industry. My job was to develop computer simulations that would use the customer's own data to demonstrate that our new tester (which cost twice as much as any other on the market at the time) would find more faults than Brand G. The models worked fine and it was the most successful product in the division's history. (Eventually the division ended up buying Brand G.)
That market pitch -- we find more faults -- was a classic example of the view that testing was the True Path to Product Quality. Alas, it was bad timing. A decade later all my simulations were in the scrap heap. That's because if the sales guy went in and tried to use that pitch the customer would probably throw him (or her by '90) out the window. A decade later the only sales pitch that worked for ATE was that the same tester was now a process monitoring tool. That's because today's paradigm in manufacturing is defect prevention. One only uses a tester to monitor the process to ensure that the process is not inserting defects into the product at an unacceptable rate. That is, one builds in the quality rather that testing it in.
Alas, the software industry is the only major industry today that has not figured this out. Today we are still dominated by the notion that if one does more testing one will ship a better product. That notion has worked for software because it is true -- up to a point. With pure testing one can get to something approaching 5-Sigma quality (0.2 defects/KLOC). However, by today's manufacturing standards, product shipped at 5-Sigma is crap. That is, the software industry has gotten away with this whole Mystique where somehow software is so difficult to do that one should expect crappy quality. As I pointed out in the previous post, it would be a very poor strategy to assume the industry can continue to get away with such nonsense.
So why is defect prevention better? The main reason, in the context of software, is simply complexity. To exhaustively test even a modestly sized program today is simply infeasible. To run the necessary test cases would take a lifetime to execute on a mainframe, much less the time to encode the test cases. That was supported by research done at AT&T and other places in the '60s and '70s. One important finding is that the number of escapes (defects that are not caught by testing) are directly proportional to the number of defects that were caught by testing. It should be clear where that is heading: if there are fewer defects in the product to begin with, there will be fewer defects that escape for any given level of testing.
In the premier shop I referred to in the previous post we were heavily into defect prevention. But before we went there the mean time to system crash was about 40 minutes and we often had hundreds of customer bug reports outstanding for each release. After two decades of militant defect prevention we got an award for 600 systems in 24x7 operation that had an MTTF of 2.7 years -- and those were mostly hardware problems! It was also unusual to have even a dozen open bug reports, even including those from our own QA group.
In doing defect prevention our testing was to monitor our process. However, the unit test suite executed overnight and the full regression suite took several days to execute -- just for a 200 KLOC device driver. There is an irony in defect prevention in that one tends to do a lot more testing. That's because one needs a decent sample of defects of each type tracked to apply statistical measures the evaluate the process. (Finding no defects is bad because one can't prove that the testing itself wasn't flawed; when we got an unusually low number of defects we usually wrote more tests.) The better the defect prevention, the fewer the defects caught so one has to do more testing simply to get sufficient samples for statistical analysis. So once one gets good at defect prevention one must also get good at testing. As a result I tend to be bemused when developers claim they can run all their unit tests "in a few minutes". In my opinion, that is just plausibility testing suitable for proof-of-concept and prototype development.
However, there is another aspect of complexity that is particularly relevant to software. Developers usually have no clue how to design good tests for large applications. Bob Binder has written an enormous tome on testing object oriented software; it is the biggest single book on the shelf in the computer section at the bookstore. How many developers have read it cover-to-cover? I haven't and I worked in the the test business for two decades. How many developers refer to it regularly when writing a test suite? How many developers have committed its more important techniques to memory? I'm betting not a whole lot. The reality is good testing is a complex discipline on its own that requires special skills -- some of which may even be inherited. The typical software developer isn't even in the same league with a professional QA person of just average abilities.
There is a more philosophical justification for defect prevention. When a program is not correct, it is broken in some fashion. The failure is just a symptom of being broken. So fixing the defect is really just addressing the symptom. Defect prevention goes to the root cause of the quality problem by addressing the why the defect was there in the first place.
My favorite example of defect prevention dates from my procedural development days just before we converted to OO development. At that point we generated very detailed Functional Specifications from the requirements as part of our analysis phase. That spec was so detailed that a line in the spec typically resulted in 1-3 lines of C code. In other words, by the time the spec was fully reviewed all of the programming issues had been resolved and coding was a rote exercise. On one process improvement cycle our Fat Rabbit was lines in the spec that simply did not get into the code at all; the developer literally skipped them. In analyzing the data it turned out that everyone had about 8-12 of these per hundred spec pages except for one developer who always had 0. We asked her what she did that might be different from everyone else. It turns out she used a highlighter pen to mark each spec line as she implemented it. So we were able to eradicate a significant number of defects by simply making using a highlighter a standard process.
The example is interesting because of the way it demonstrates how defect prevention and process improvement works. However, it is even more interesting from a cultural viewpoint. Note that we collected data on individual developers and analyzed it publicly. Very few shops today have the cultural mindset to support this sort of thing. One has to have a process improvement culture in place that is focused on solving problems rather than allocating blame. Equally important, the developers have to be completely on board with the notion of self-improvement as a characteristic of professionalism. That is, the developers must want to improve their performance. Then the data and the multiple reviewers are just a means to a desirable end.
Blog root page
previous page in category
nextpost in category
