H.S. Lahman on OO and MDA

1.0 Testing and development process

Blog root page
next post in category

The state of software quality is pretty abysmal today because the industry is too new and too arrogant to do what is required to provide better software quality. There are many measures of software quality: correctness, whether it does what the customer wanted; completeness, whether it does everything the customer expected; reliability, whether the software is available to do specific tasks for the customer; maintainability, whether the software is easy to modify and enhance; and usability, whether the software fits seamlessly into the customer's environment. And these are just the ones that tend to get the most press.

The key lesson that the software industry has not learned yet is that whether the software quality is "good" or not with respect to any of these views depends on the processes that surround the intellectual activities of design and coding rather than on the activities themselves. With the partial exception of maintainability, the quality of the software does not depend at all upon the intellectual content that the developer invests in the software product. Instead, it depends almost entirely upon the way the developer provides the intellectual content.

Of all the things we do as software engineers, the least important is writing programs.

Sadly, too many young people come to software development believing that writing 3GL code is what software is all about. The end product of our endeavors is, indeed, a computer program. However, in the end all that program does is move bits from one pile to another. We will be judged as professionals based upon the quality of that program. The quality of that program will depend not on what we wrote but how we wrote it.

In the '80s I had the privilege of working in a premier software shop doing conventional procedural development. It was a premier shop because it produced software with nearly 5-Sigma defect rates* even though it was a commercial software shop. As important, the shop built that software with productivity that was integer factors better than published data for similar software from places like MS and IBM. The shop routinely collected data on virtually every developer activity. One of the more interesting statistics was that developers spent only about 5% of their time actually writing 3GL code to the point where it compiled without error. Nearly 40% was pure overhead; paid time engaged in activities not directly related to a specific development project. The other 55% of their time was spent on all those niggling little processes that surrounded the coding. So actually mucking in 3GL code occupied less than 10% of their project time.

Most shops spend a whole lot larger fraction of their project time writing code. And they are doing something wrong because I'll give substantial odds they don't come close to the combination of defect rates and productivity that we provided. Getting all that peripheral process stuff right is what enables a shop to be a premier shop. A by-product of that is that one spends a lot less time on coding and a lot more time on thinking.

For the rest of this category I am going to focus on the correctness view of software quality because that is where I believe there will be a major disruption in the industry over the next decade. The reason for my assertion is that software users are beginning to figure out that the only thing that is broken in their lives today is software. The software industry could get away with poor quality for many years because software was perceived by others as an arcane discipline not unlike alchemy. But now software is as ubiquitous in people's lives as automobiles and household appliances. So the users of software see a very stark contrast between software and everything else. And they see that contrast constantly. And they see that quality contrast primarily in terms of correctness and reliability.

That means that defects are no longer acceptable as a by-product of some mystical wizardry. Users are starting to demand that software provide the same quality as everything else. Unfortunately there aren't many competitive alternatives right now because almost all popular commercial software is crappy. So the pressure is only showing up indirectly, as in a rapid increase in product liability lawsuits that have demonstrated that the "as-is" warranty disclaimers for software aren't worth the paper they are written on. [Note that the recent UCITA initiative by the software industry was aimed at putting legal roadblocks in the way of consumer redress. But that's another story...]

The lack of alternatives, though, presents an enormous opportunity for competitive advantage. When an alternative product with better quality does show up in the market, it will have a huge competitive advantage. More important, that competitive advantage will last awhile, just as it did in the '80s when the PacRim kicked the crap out of Western manufacturing paradigms, because it takes a big process investment to catch up. This is exactly what has happened to WinX vs. Linux. Linux got unexpectedly rapid market share because it initially had better security and reliability. That forced MS to play catch-up in a major internal effort over the past several years. Even now, when MS has pretty much caught up, the market perception is that WinX is insecure and unreliable compared to Linux. That has cost MS some very serious money over the past few years.

The moral is: Don't be the one playing catch-up on software quality. The next few category posts go into more detail about what needs to be done.

* Sigma is a nonlinear scale for measuring software defect rates. 5-Sigma is roughly 0.2 defects per KLOC. This was the level for mission-critical software in the '70s and '80s; currently mission-critical software is pushing 6-Sigma (~0.01 defects/KLOC). The industry average for commercial software is currently slightly better than 4-Sigma (~2 defects/KLOC).

next post in category
Blog root page

2.0 Testing vs. defect prevention

Blog root page
previous page in category
next post in category

The was a "quality revolution" in the '80s for manufacturing of all kinds.  It was led by the Japanese but it was closely followed by the rest of the PacRim.  Previously the conventional wisdom was that one could "test in" quality by finding defects and removing them (either fixing the defect or not shipping the part).  The quality revolution changed all that by demonstrating that quality is a process issue and testing is inadequate for ensuring correctness or reliability in a product.  That was demonstrated in the marketplace when PacRim manufacturers grabbed huge chunks of market share from traditional Western manufacturers simply because their products worked while the others' didn't.

That revolution profoundly changed the way Western manufacturers viewed quality and they had to d a whole lot of scrambling to scrap the old paradigm and install the new one.  However, by the '90s the new paradigm was firmly ensconced throughout all manufacturing activities.  It even migrated into service industries and pure engineering disciplines.

A relevant anecdotal data point.  In '79 I was hired by a company that made Big Honking Automated Test Equipment (ATE) for the electronics industry.  My job was to develop computer simulations that would use the customer's own data to demonstrate that our new tester (which cost twice as much as any other on the market at the time) would find more faults than Brand G.  The models worked fine and it was the most successful product in the division's history.  (Eventually the division ended up buying Brand G.)

That market pitch -- we find more faults -- was a classic example of the view that testing was the True Path to Product Quality.  Alas, it was bad timing.  A decade later all my simulations were in the scrap heap.  That's because if the sales guy went in and tried to use that pitch the customer would probably throw him (or her by '90) out the window.  A decade later the only sales pitch that worked for ATE was that the same tester was now a process monitoring tool.  That's because today's paradigm in manufacturing is defect prevention.  One only uses a tester to monitor the process to ensure that the process is  not inserting defects into the product at an unacceptable rate.  That is, one builds in the quality rather that testing it in.

Alas, the software industry is the only major industry today that has not figured this out.  Today we are still dominated by the notion that if one does more testing one will ship a better product.  That notion has worked for software because it is true -- up to a point.  With pure testing one can get to something approaching 5-Sigma quality (0.2 defects/KLOC).  However, by today's manufacturing standards, product shipped at 5-Sigma is crap.  That is, the software industry has gotten away with this whole Mystique where somehow software is so difficult to do that one should expect crappy quality.  As I pointed out in the previous post, it would be a very poor strategy to assume the industry can continue to get away with such nonsense.

So why is defect prevention better?  The main reason, in the context of software, is simply complexity.  To exhaustively test even a modestly sized program today is simply infeasible.  To run the necessary test cases would take a lifetime to execute on a mainframe, much less the time to encode the test cases.  That was supported by research done at AT&T and other places in the '60s and '70s.  One important finding is that the number of escapes (defects that are not caught by testing) are directly proportional to the number of defects that were caught by testing.  It should be clear where that is heading: if there are fewer defects in the product to begin with, there will be fewer defects that escape for any given level of testing.

In the premier shop I referred to in the previous post we were heavily into defect prevention.  But before we went there the mean time to system crash was about 40 minutes and we often had hundreds of customer bug reports outstanding for each release.  After two decades of militant defect prevention we got an award for 600 systems in 24x7 operation that had an MTTF of 2.7 years -- and those were mostly hardware problems!  It was also unusual to have even a dozen open bug reports, even including those from our own QA group.

In doing defect prevention our testing was to monitor our process.  However, the unit test suite executed overnight and the full regression suite took several days to execute -- just for a 200 KLOC device driver.  There is an irony in defect prevention in that one tends to do a lot more testing.  That's because one needs a decent sample of defects of each type tracked to apply statistical measures the evaluate the process.  (Finding no defects is bad because one can't prove that the testing itself wasn't flawed; when we got an unusually low number of defects we usually wrote more tests.)  The better the defect prevention, the fewer the defects caught so one has to do more testing simply to get sufficient samples for statistical analysis.  So once one gets good at defect prevention one must also get good at testing.  As a result I tend to be bemused when developers claim they can run all their unit tests "in a few minutes".  In my opinion, that is just plausibility testing suitable for proof-of-concept and prototype development.

However, there is another aspect of complexity that is particularly relevant to software.  Developers usually have no clue how to design good tests for large applications.  Bob Binder has written an enormous tome on testing object oriented software; it is the biggest single book on the shelf in the computer section at the bookstore.  How many developers have read it cover-to-cover?  I haven't and I worked in the the test business for two decades.  How many developers refer to it regularly when writing a test suite?  How many developers have committed its more important techniques to memory?  I'm betting not a whole lot.  The reality is good testing is a complex discipline on its own that requires special skills -- some of which may even be inherited.  The typical software developer isn't even in the same league with a professional QA person of just average abilities.

There is a more philosophical justification for defect prevention.  When a program is not correct, it is broken in some fashion.  The failure is just a symptom of being broken.  So fixing the defect is really just addressing the symptom.  Defect prevention goes to the root cause of the quality problem by addressing the why the defect was there in the first place.

My favorite example of defect prevention dates from my procedural development days just before we converted to OO development.  At that point we generated very detailed Functional Specifications from the requirements as part of our analysis phase.  That spec was so detailed that a line in the spec typically resulted in 1-3 lines of C code.  In other words, by the time the spec was fully reviewed all of the programming issues had been resolved and coding was a rote exercise.  On one process improvement cycle our Fat Rabbit was lines in the spec that simply did not get into the code at all; the developer literally skipped them.  In analyzing the data it turned out that everyone had about 8-12 of these per hundred spec pages except for one developer who always had 0.  We asked her what she did that might be different from everyone else.  It turns out she used a highlighter pen to mark each spec line as she implemented it.  So we were able to eradicate a significant number of defects by simply making using a highlighter a standard process.

The example is interesting because of the way it demonstrates how defect prevention and process improvement works.  However, it is even more interesting from a cultural viewpoint.  Note that we collected data on individual developers and analyzed it publicly.  Very few shops today have the cultural mindset to support this sort of thing.  One has to have a process improvement culture in place that is focused on solving problems rather than allocating blame.  Equally important, the developers have to be completely on board with the notion of self-improvement as a characteristic of professionalism.  That is, the developers must want to improve their performance.  Then the data and the multiple reviewers are just a means to a desirable end.

Blog root page
previous page in category
nextpost in category 

3.0 Unit Testing with MBSE

Blog root page

previous post in category

One of the advantages of using object state machines to describe object behavior in the MBSE approach is that they are very easy to test at the unit test level.  All one has to do is to disable the event queue manager so that it only executes the first event placed in the event queue and just invokes a logging facility in the test harness to log any event placed on the queue subsequently by the action under test.  Then to test a state action, one just pushes the relevant transition event onto the event queue.

This is easy because a state action is completely defined by three things: (A) the state of the application prior to executing the action, (B) the state of the application after executing the action, and (C) any messages the action generates (i.e., the events it places on the event queue).  The state of the application is represented by the values of the state variables (object attributes).  In a well-formed OO application a method accesses any data it needs directly so one knows exactly what attributes need to be initialized prior to execution by simply inspecting the method.  The test case specification will indicate exactly what state variables will be modified when the action executes.

Because we construct the behaviors in MBSE using an asynchronous collaboration model, we don't care what might happen in response to any events the action under test places on the event queue.  This is important because it means that we can specify an action's interaction with other objects fully in terms of the collaboration messages it creates rather than hierarchically through the specification of the resulting behaviors.  [Remember that in OOA/D collaboration messages are announcements of something the sender did rather than imperatives about what the receiver should do.  So the specification of the behavior responsibility is limited to what the behavior does and what it should announce about what it did.]

These things all conspire so that all we need to instantiate in the application, besides the action under test, are any objects containing attributes read or written by the action under test.  These objects are effectively stubs because we will not execute any of their behaviors.  This makes life very easy for the test harness because all we need is the Class Diagram definition for these objects and it is fairly easy to automate both the attribute initialization and validation of the attributes after execution.  Thus the basic unit test process is:

(1) Instantiate the application objects.  This essentially means creating an instance of the object under test and a stub instance for any objects that have attributes accessed by the methods of the object under test.

(2) Initialize the application objects.  Initialize any attributes that the object under test accesses to appropriate values.  For attributes that the object reads, choose values consistent with the specified initial conditions for the test case.  For attributes that the object writes, choose values that are different than those specified as outputs by the test case.

(3) Initialize the current state of the object under test.  This should be to a state that is appropriate to the transition associated with the test event.  (The test should specify the state the state machine is expected to be in for a specific unit test case.)

(4) Push an event onto the event queue.  This event is specified in the test case to invoke a particular state action.  If the event queue implementation requires that the queue be formally started, do so to execute the event.

(5) Validate the application state.  Examine the state of the application.  Essentially this means validating that any attributes the action writes have the value specified in the test case.

(6) Validate the collaborations.  Examine any events generated by the action to ensure the correct event ID, object address, and data packet were supplied as specified in the test case.

Caveat: this scheme only does positive testing (i.e., it validates that the action does what its responsibility was specified to do).  It does not demonstrate that the action does not do something it shouldn't (i.e., that it does not update an attribute that it is not supposed to update).  One can do negative testing but it entails instantiating every possible object that might be created during the execution, initializing all their attributes to some flag value, and then validating that the flag values did not change when the action under test executes.  Then one must repeat the same test with a different flag value (in case the action under test coincidentally writes exactly the flag value).  [Even this is not fool-proof since the action under test may simply access one attribute it shouldn't and then write that value someplace else it shouldn't.  Negative testing is always difficult but this unit testing approach covers most of the bases.]  Whether such negative testing is feasible is an economic issue for the development environment, but this approach gets one close to exhaustive negative testing when it is justified.

This process represents pretty vanilla test execution, so it should be fairly easy to automate from a test harness.  For elaborated code there is some trickiness involved in setting the current state of the state machine since typically the current state is a private attribute with no public interface.  In a language like C++ one can make the test harness a "friend" and provide private access that way.  Otherwise one needs to provide a public setter for it that only the test harness accesses.  (Since public vs. private at the 3GL level is irrelevant in translation-based development, one typically provides a setter as a matter of course.)

It is important to note that this ease of use is enabled by good OO practices that are religiously followed.  That's because the rules of state machines enforce good OO practices, such as self-contained methods.  One can do the same thing without state machines but it requires substantial discipline that is often lacking in OOP.  For example, all behaviors must be procedures rather than functions so that they can be fully stubbed without affecting the specification of the object under test.  In addition, any called methods cannot modify attributes that the caller accesses after the call.

Blog root page

previous post in category