H.S. Lahman on OO and MDA

1.0 The basic OO process

Blog root page
next post in category

OO development is a rather unique approach to software development that emphasizes problem space abstraction. It is less intuitive than procedural or functional programming approaches that are based directly upon the organization of computers and the computational models of Turing and von Neumann. However, its close mapping to the customer problem space tends to provide a long-term stability for applications. In addition other techniques combine with problem space abstraction to produce more maintainable software.

The primary difference between OO development and procedural or functional development lies in the role of sequences of operations in the development. In procedural or functional development one focuses initially on such sequences and decomposes them into program units. While the specific mode of decompostiion varies, the development is still driven by analyzing sequences of operations.

In contrast, OO development begins with creating a static view of the problem space. That view is elaborated with more detail until one has a complete skeleton for the application. Only then does one deal with the dynamics of the solution and then it is highly localized. Only in the final step does one deal with the specific flow of control of the overall problem solution. Fundamentally the OO approach answers five questions in order:

(A) What problem space entities are involved in the solution to the problem in hand?

(B) What intrinsic responsibilities do those entities have that are necessary to solve the problem in hand?

(C) What logical relationships to the entities have with each other on a peer-to-peer basis?

(D) What do the entities actually need to do to solve the problem in hand?

(E) What are the collaborations between entities that are necessary to solve the problem in hand?

Thus the basic steps in OO development approach are:

(1) Identify the the problem space entities that are relevant to the problem solution. While doing so we relate similar entities into sets that we identify as classes. Basically this means providing classifier boxes on a UML Class Diagram. The key mapping here is that the entities must be readily identifiable in the problem space. They may be conceptual rather than tangible, but they must be readily recognized by any domain expert.

(2) Identify the intrinsic responsibilities each entity has that are relevant to the problem in hand. In OO development responsibilities are defined in terms of knowing something (knowledge) and doing something (behavior). In this step one does not care how they know or do these things; one just identitifies the responsibilities. Very commonly behavior responsibilities are anthropomorphized as entity roles. That's because software replaces things that people do but we rarely model people directly; instead we model things and concepts in the problem space. So we must allocate the things that the replaced people would do as behavior responsibilities of the entities identified in (1). This is also done on a UML Class Diagram.

[This is perhaps the most fundamental difference between OO and procedural/functional development. In procedural/functional development one goes directly to what people do and captures those activities as procedures. Procedures are then collected into program units that roughly correspond to the people in the problem space. Those units are then named with the role the person person plays in the solution, such as XxxManager or XxxController. In the OO approach one only abstracts entities that are tangible or conceptual outside the context of individual people. That's because such entities are more stable than people's roles so they provide a more stable long-term architecture for the application.]

(3) Identify fundamental relationships between sets of entities (classes). These relationships define the structure of participation in interactions between members of the sets. In effect, they place constraints on what members of another set a member of the set in hand can interact with. That is, relationships define very basic static (invariant) limits on participation in collaborations. This is also done on a UML Class Diagram.

(4) Define the intrinsic behaviors of the entities that resolve their behavior responsibilities. The operative word here is intrinsic. One defines what the entities do without direct regard for the overall solution context. In particular, individual behavior responsibilities are resolved without any dependence upon other entity behaviors. In effect the step is about abstracting behavior in a manner that is tailored to the problem in hand without undue consideration of specific solution flow of control. This will be done with UML Statecharts and/or synchronous methods where the actual activities are described with an abstraction action language.

(5) Define collaboration messages. Essentially this connects the behavioral dots by defining who sends messages and who receives them. The sequencing of messages defines the overal problem solution's flow of control. In UML this is typically done on one of the Interaction Diagrams.

Blog root page
next post in category

2.0 Identifying objects; blitzing

Blog root page
pervious post in category
Next post in category

There have been many proposals for identifying objects, ranging from Coad's extraction of nouns from the SRS to Wirfs-Brocks extraction of roles from the problem space. In my experience the best way to define objects is through an object blitz. The basic object blitz process is described below. Any or all of the various methodologists' techniques can be employed in an object blitz for identifying object candidates.

The reason I prefer blitzes is because it is a team effort. I strongly feel that identifying problem space entities to abstract should be a team effort because it merges multiple views of the problem space and the exercise itself is a strong tool for creating a common consensus on problem space semantics among the team members. In fact, I regard the object blitz as more of a consensus-building exercise than a formal design activity.

However, to be effective the blitz must be a disciplined activity. It is precisely because different team members will have different perspectives that care must be taken to avoid pitfalls like lengthy digressions and personal views. Therefore it is imperative that the blitz be led by someone familiar with the process who can guide the discussions effectively. It is also important to time-box some of the activities. The process described below is based largely on well-established techniques for consensus building.

The ideal setting is a large conference room with a large whiteboard. Several pads of 3x5 Post-Its and a bunch of pencils will be necessary. A good eraser is also important. A side easel with a pad (aka the Parking Lot) to record deferred issues is handy but any recording medium will do. The basic steps are:

(1) Explain the ground rules. This is for the benefit of anyone who has not done a disciplined blitz before. The basic process and the rules are briefly described.

(2) Identify preliminary candidates. This is a free-form exercise and it is the core blitz activity. People offer stream-of-consciousness suggestions for entities to abstract. Each suggestion is recorded on the whiteboard. There should be no discussion and each suggestion should be accepted without prejudice. It is important that people not be inhibited when offering candidates so accept everything, no matter how silly or pointless. The entire exercise should take no longer than fifteen minutes and it will typically start to run out of steam after 6-8 minutes.

(3) Preliminary screening. The goal here is to eliminate duplicates or candidates that are clearly irrelevant, attributes of other objects, and whatnot. This is just a screening so if there is any doubt the candidate remains. It is also a free-form exercise in that anyone can suggest removing a candidate. Any discussion should be limited to no more than a minute per object; if there is no consensus to dump it, it remains. The survivors are each written on a Post It and tacked to the whiteboard.

(4) preliminary binning. Each of the remaining objects is evaluated one at a time. Whoever suggested it provides a sentence or two to describe the basic semantics of what the entity is and how they feel it is relevant to the problem in hand. The goal is to bin the object into one of four categories: Essential, Probable, Unlikely, and Removed. The categories represent how important the object is to the problem in hand. The goal is not to get the semantics right yet; this is just a prioritization process. Any discussion should be limited to a minute or so. If consensus can't be reached between two bins, put it in the higher importance bin. The whiteboard is divided into the three survivor bins and the Post Its are put in the appropriate section.

[A very viable alternative is for the team to do the binning. Each person takes a small pile of Post Its and bins them. The team then mills around the whiteboard to validate the binning. If someone wants to move a Post-It from one bin to another they should state aloud that they are doing so and why. If there is consensus, it is moved. The discussion should be time-boxed as above with the same tie-break.]

(5) Semantic definition. Whoever suggested each surviving candidate fills out 1-3 sentences on the Post-It to describe their view of object semantics. The Post Its are then returned to their bins. This is done in parallel and usually takes about 15 minutes. Do not describe what it does or what it knows except in the most general terms; in this exercise one describes what the underlying problem space entity is. This sort of description is often called a mission statement.

(6) Scrubbing. When all the Post Its are done they are reviewed one at a time, starting with the highest priority bin. The author's definition is read by the moderator and the team comments on possible changes. Time-box discussion to 5 minutes for highest, 3 minutes for Probably, and 1 minute for Unlikely. The Post-It is modified by the moderator as consensus is reached. If consensus cannot be reached, it is moved to the Parking Lot.

(7) Deal with deferrals. Issues may be raised about the actual requirements and the problem space during discussions. These go on the Parking Lot along with the objects for which consensus could not be reached. If time allows, some or all of them may be dealt with at the end of the blitz. Priority should be given to resolving object semantics. If all the deferred items cannot be dealt with, another meeting is scheduled. There may also be action items for the deferred items resolution before the meeting.

IMO, the most effective way to resolve lack of consensus on object semantics is to have a domain expert present during the blitz. When in doubt defer to the domain expert's view of what the entity is. If a domain expert can't be present, there should be a means for contacting one to review the contentious objects before the next meeting. In those very rare situations where a domain expert is not available, some other mechanism for conflict resolution will have to be defined before executing the blitz. That is, the blitz must be able to resolve all the issues somehow before proceeding further with the design. More important, that mechanism has to avoid acrimony.

Note that there was no mention of identifying knowledge and behavior responsibilities. Trust me on this that doing the object semantics is more than enough for one exercise. This is, at heart, a creative process and one does not want the team to be "stale" when defining the basic building blocks of the application. So leave defining object properties as a separate exercise.

Blog root page
previous post in category
next post in category

3.0 Agile Processes

Blog root page
Previous post in category
next post in category

Agile processes are getting a lot of press lately.  It is important to realize that there are actually two main camps in using agile processes.  The best known one is for OOP-based agile processes like XP and Scrum.  These processes are highly focused on writing 3GL code and any OOA/D is highly distilled into practices like system metaphors and militant refactoring.  They also incorporate a very fine-grained development life cycle model for Incremental, Iterative Development (IID).  In these processes any modeling at the UML level is strictly optional and throwaway.  These processes rely on applying very specific programming practices within a very formal process.  These processes tend to be very rigid because they dictate developer activity down to the code fragment level and they are very specific about exactly what programming practices will be employed and how they will be employed.  (As a group, the OOP-based agile processes are among the moos detailed and rigid processes that I have ever seen in software development; they are rivaled only by the operational standards for occupations like telephone repairman.)  In effect they represent the culmination of decades of hard-won experience with writing 3GL code.

At the other end of the spectrum are the model-based agile processes that are based upon translation.  In translation one employs a true 4GL solution representation, such as UML, to resolve functional requirements and relies on a tool to automate optimization of nonfunctional requirements.  These approaches emphasize design reuse through militant automation to address many 3GL concerns.  In effect, they represent a paradigm shift away from 3GL programming comparable to the paradigm shift of the '60s that moved from Assembly programming to 3GL programming.  Hybrid approaches, such as Ambler's Agile Modeling, lie in the middle.

Both approaches represent valid evolutions of software engineering but they have evolved in quite different directions to the point that they are almost impossible to compare on a feature-by-feature basis.  However, one can compare them at the megathinker level based on how they solve various software development problems, which I will attempt below.  In that comparison I will employ XP as the post child for OOP-based agile processes and Model-Based Software Engineering (MBSE) as the poster child for translation processes.

Requirements gathering.  Requirements gathering in XP is quite informal; often verbal in nature.  XP depends upon defining requirements for one functional increment (usually on the order of a couple of programming days of effort) at a time and having a customer available to clarify the requirements on at least a day-to-day basis.  This is a logical outgrowth of fine-grained IID and a strong belief that only the customer can really define requirements (e.g., throwing requirements "over the wall" in a formal specification just invites misinterpretations).  While having the customer intimately involved in the development at all times is clearly a good way to ensure the right requirements are properly implemented, it is also an Achilles Heel for agile processes.  If the customer (or a surrogate, such as a marketing representative) is not readily available on a daily basis, the informal nature of requirements gathering will tend to fall apart.  That places a major constraint on the business environments where processes like XP can be deployed.

In MBSE the requirements can be supplied either in terms of formal specifications (use cases, etc.) or informally in the form of discovery of the problem space with direct customer involvement.  The former is common for large, distributed projects while the latter is common for projects at the scale of 5-20 person teams.  Validation depends upon the fact that the solution only deals with functional requirements and the solution itself is abstracted solely in customer terms.  (The PIM model should be fully implementable without change in the customer's environment as a manual system, however inefficient that might be.)  In theory this allows the customer to be more directly involved with the actual solution because of the abstract nature of the UML representation.  As a practical matter, though, there tends to be a significant learning curve for that level of customer participation beyond the static models.

Development Life Cycle Model.  In the OOP-based agile processes IID is mandatory.  More important, the granularity is rigidly defined and increments are quite small.  An increment is typically three calendar weeks and individual "stories" specified within that increment are usually 1-4 days.  The maximum size of increments is mandated and the various activities (e.g., estimation, testing, etc.) are strongly tied to small increments.  A key tenet is that whatever is constructed in an increment must be executable so that it can be fully tested for acceptance by the customer.  A key goal is to avoid the classic "90% done" syndrome that plagued large projects in the SA/SD era.

While the translation processes are quite amenable to IID, it is not mandatory so they could be applied in the traditional Grand Waterfall manner.  However, the "waterfall" is quite different because the "design" and "implementation" stages of the classic software waterfall are completely automated by the transformation engine.  Unfortunately a common misconception about model-based processes is that all the models for the project must be created "up front".  (BUDF = Big Design Up Front.)  This is not true at all.  All one need to model "up front" is the minimum needed to execute something, just as in the OOP-based processes.  As a practical matter one actually has to model a lot less than a corresponding OOP-based increment actually implements because one is only concerned with functional requirements in the model.  That is, all the "boilerplate" to address nonfunctional requirements properly is handled by transformation engine automation.  The bottom line is that executable MBSE models can be created at the same or even smaller increments than in OOP-based processes.  In other words, the development life cycle model is completely customizable.

Testing.  Testing is an integral part of OOP-based agile processes.  This is highly laudable given the sad state of traditional developer testing.  Both test-first and test-driven design are highly desirable process elements.  However, processes like XP employ testing as more than just an important element of the process; it is actually central to the development philosophy.  For example, in XP the only formal expression of requirements are the tests provided by the developers and customers; if they pass, then the software satisfies the requirements by definition.  In my opinion, this has two pitfalls.  The first is that it assumes developers and customers can write good tests, which is unlikely at best in my experience.

[There is also a resource issue.  Writing acceptance tests for a 10-person XP team would require roughly two customers working full time (Ron Jeffries; private communication).  One can substitute an external QA team of professional test writers as a surrogate for the customer here.  However, that leads to another problem: synchronization. How does one ensure that both the QA team and the developers are getting the same requirements, given the very information process for eliciting requirements on a verbal, ad hoc basis?]

More important, it is tied to the traditional view that product quality can be assured through "testing out" defects.  That view was dismissed in other engineering disciplines in the '80s when the PacRim nations revolutionized product quality based upon defect prevention.  Testing can bring one close to 5-Sigma reliability but to get to 6-Sigma and beyond one must practice militant defect prevention where testing becomes a tool for process monitoring rather than ensuring product quality.  The XP view of testing, when combined with its already very rigid process structure, concerns me because it tends to stifle process improvement directed at preventing defects.

Alas, the model-based processes do not have any built-in policy or infrastructure for testing.  The translation processes do allow early testing of the solution models for functional requirements because the models themselves are executable.  Because the translation processes emphasize good application partitioning with disciplined subsystem interfaces, one can also conveniently do full functional testing of subsystems.  But the amount, quality, and goals of testing are left as an exercise for the development environment.  On the other hand, I have to point out that statistically the reliability of translation applications tends to be substantially better than applications built via elaboration (i.e., manually coded at the 3GL level).  I attribute that to focus on functional requirements, economies of scale in design reuse, and the compactness of a 4GL representation.

Dependency management.  Most of the refactoring in OOP-based processes is focused on dependency management (simplistically: ensuring implementation dependencies form a directed, acyclic graph).  This is driven by two considerations.  One is to apply OOA/D principles (e.g., encapsulation, implementation hiding, etc.) to the development.  Much of the content of Fowler's book, Refactoring is a distillation of good OOA/D into "cookbook" practices at the object and code fragment level.  Unfortunately the other driving factor lies in dealing with physical coupling in the OOPLs.  The OOPLs do a fine job of minimizing logical coupling but a pretty awful job on physical coupling (i.e., what one compilable unit needs to know about another).  Most of dependency management is focused on the physical coupling problem.  That means that a lot of the development activity at the 3GL is focused on solving a developer problem -- code maintainability -- rather than the customer's problem.  (Note this is a general 3GL development problem; the OOP-based processed just incorporate a solution explicitly.)

In the translation processes 3GL physical coupling is completely irrelevant because one never has to maintain the code directly.  Instead, any changes are made to the models and the relevant code is regenerated.  In addition, the level of abstraction (OOA) of the models is quite high and the focus is on abstracting the problem space correctly.  So, unlike the OOP-based processes where refactoring begins after passing the tests, one simply stops modeling when the tests pass.  Thus refactoring is usually only done to ensure correctness in resolving functional requirements (i.e., "fixing" bugs).  That is, one employs good OOA practices at a high level of abstraction to ensure that the problem space is captured correctly and those practices ensure long-term maintainability at the model level.  (I would also argue that the OOA practices employed are the purest form of of OO development because they are unsullied by the compromises that the OOPLs make with the hardware and the computational models of Turing and von Neumann.)

Blog root page
previous post in category
next post in category

4.0 Basic process improvement

Blog root page
previous post in category
next post in category

Probably the most important activity in software development is process improvement.  The technologies and tools for software development change very rapidly and that means that the processes around those technologies and tools need to change as well.  In addition, process improvement is absolutely crucial to defect prevention (See the category on testing and defect prevention).

Unfortunately the industry has had remarkably bad experiences with ad hoc process improvement.  Process changes get lost so we repeat the mistakes of the past.  Changes become obsolete so we waste effort doing irrelevant things.  Processes "creep" over time because they are not properly documented and developers are not properly trained in them.  Perhaps worst of all, ad hoc process changes are not properly integrated into the development environment so that they are inefficient and bureaucratic.  As a result ad hoc process improvement simply doesn't work over the long term.

One problem is that software developers generally don't know very much about process engineering, which is a discipline that is comparably complex as software engineering.  Many years ago in my naive youth I was a member of a team assigned the task of documenting our software development process.  We struggled mightily for three weeks and came up with a 25-page opus.  With 20/20 hindsight I now realize that document was complete crap.  For example, the first 2 pages described the process and the remaining 23 pages described all the exceptions.  That wasn't a process; it was anarchy.

For process improvement to be effective, it needs to be have several basic characteristics:

Incremental.  Developer resources are scarce and developers cannot be sucked into a rabbit hole of effort trying to solve the process equivalent of World Hunger.  Process improvement needs to be narrowly focused and "effort-boxed" so that it can be integrated painlessly into the developer's main job of producing software.

Disciplined.  Process improvement also needs to take advantage of lessons others have learned.  So the way one does process improvement should already incorporate good practices from other disciplines like process engineering.  That is, the process discipline already formally incorporates proven practices learned in the past and in other contexts.

Systematic.  To be efficient, one needs to go about process improvement without having to grope for solutions to problems that recur in different forms.  One also needs to take good advantage of any tools that are available.  Basically that means that many of the process improvement activities should be ingrained in developers as a matter of habit.  Developers should be able to "walk" the path to process change in very much the same way during every increment of improvement.

Simple.  The notion of KISS that is so cherished by software developers also applies to process improvement.  One of the <many> problems with the process description in my example above was that it was far too complicated.  Good process descriptions have less than eight steps, each defined in a single sentence.  If more detail is required, individual steps are decomposed as new processes at a lower level of abstraction.  And one quits decomposing process steps when one cannot reach consensus on a single set of steps for the process.

[Actually, process descriptions can have substantial supplementary information.  For example, the supplementary information could include inputs (prerequisites) that must be in place to execute the process.  But that material is peripheral to the actual process; it is simply optional documentation.  The process steps themselves should be clear enough to execute for anyone familiar with the general process context.  And any supporting documentation should be limited to what the developers themselves feel is important.]

Communicated.  Obviously process changes need to be communicated to everyone affected and any new personnel need to be properly trained.  In addition, the process of doing process improvement needs to be communicated in the same fashion.  [Note that I avoided the term documented, which tends to conjure up images of huge piles of paper.  Though rare, verbal communication can be adequate in some contexts.  The point here is that the form of the communication doesn't really matter; whatever works.  In addition, KISS applies to the communication media as well.]

Data collection.  Good process improvement is impossible without hard data.  However, it is extremely important that data collection be minimally intrusive to ensure effort is not diverted from software production and process improvement.  Therefore as much care should be devoted to providing good data collection systems as is devoted to providing good process improvement.  The two should be regarded as being inextricably linked.

Probably the greatest pitfall in doing process improvement is ignorance.  As I indicated above, software developers rarely know anything about process engineering.  (A sad commentary on the state of CS education, in my opinion.)  That's why the alphabet soup of process frameworks exist: CMM, TQM, ISO, et al.  Such frameworks distill the academic discipline of process engineering into standards, guidelines, and other infrastructures that support process improvement in software development.

[Unfortunately, the second greatest pitfall in process improvement is taking such frameworks too literally.  For example, the CMM has a horrific reputation among software developers for bureaucracy because of a few misguided implementations.  In reality the CMM just defines what one needs to do, not how to do it.  But too often shops look at examples drawn from DoD projects with a cast of thousands spread over three continents and then apply them literally to a team of a dozen developers with disastrous results.  But that is fodder for another suite of posts...]

In the next couple of posts I will describe some very basic processes that can be employed for systematic incremental process improvement.  Practicing these two processes will go a long way towards implementing process improvement in a development environment.  But in the long term one really needs a full process improvement framework.

Blog root page
previous post in category
next post in category

5.0 7-Step process improvement process

Blog root page
previous post in category
next post in category

In this post I present a very basic process for incremental process improvement based upon TQM's "7-Step" process.  The goal of this process is to identify a process change that is necessary to eliminate some problem in the development space.  It provides a very simple, systematic approach to doing process improvement in an incremental fashion that minimizes developer effort when achieving some measurable improvement in the way the software is developed.

Typically this process is executed by a small team (3-8 people) conventionally known as a Quality Improvement Team (QIT).  The QIT is formed to address a particular problem and then the QIT executes the 7-Step process.  Not unexpectedly the process has seven steps:

(1) Identify the problem to be solved.  The objective of this step is to make sure the QIT understands exactly what the problem is.  If it turns out the problem is too large to be addressed in a single increment, then the QIT uses this step to try to prune the scope of the problem to something that is more manageable.  If the problem looks like it will take more than about 16 hours of effort per team member to solve it, then it it is probably too large for a single QIT increment.  A good rule of thumb is that developers should spend about 10% of their time on process improvement.  Another rule of thumb is that at QIT should complete its work in 2-4 weeks.

(2) Collect and analyze data.  In this step the QIT collects hard data on the problem.  For example, suppose the problem is that there are two many escapes in the software after testing.  Data is required to isolate the defects that are the best candidates for proactive defect prevention.  That data might include defect frequency and repair cost to determine the defect type with the "biggest bang for the buck".

(3) Root cause analysis.  To eliminate the problem one needs to clearly understand how and why the problem occurs.  This step is designed to provide insight into what the real underlying cause of the problem is based upon the data from (2).

(4) Plan and implement a solution.  In this step the QIT evaluates alternative solutions that might address the problem and selects the one that seems most practical.  (The notion of practical includes both the effectiveness of the solution and the cost of solution; a sort of cost/benefit analysis.)  The solution is then implemented on an experimental basis.

(5) Evaluate the the solution.  The basic idea here is to be sure one doesn't introduce process changes that are not effective or that are otherwise impractical.  That is, the QIT needs to ensure that the benefit outweighs the cost in practice as well as theory.  If things don't work out as hoped, go back an appropriate previous step.

(6) Standardize the solution.  Here the QIT rolls out the process change for integration into the production environment.  Mostly this is about communicating the change to everyone it affects and ensuring that it becomes part of the normal development process.  The specific things to be done are very dependent on the environment and the change.  For example, communicating it to everyone can range from sending an eMail to providing a two-day training class.  Another key goal here is to institutionalize the change so that it doesn't get lost.  Typically that involves permanent documentation in some central process repository.  Again the particular infrastructure will be tailored to the development environment.  Basically the QIT determines the best way to roll out the change and then does so in this step.

(7) Reflect on the process.  This is a self-correcting mechanism for the 7-Step itself.  The QIT evaluates its own performance to determine things like how effective the data collection was and whether the tools employed at each step were effective.  This provides insight to the QIT members that they carry forward to other QITs to make them more efficient.  If the QIT uncovers something important (e.g., a new tool to use in a step), that can be directly communicated directly to the other developers in this step.

This is a very cursory description.  Among other things, I didn't mention any of the tools that can be used at the various steps.  For example, in Step (1) there are standard (but optional) tools for gaining consensus (e.g., K-J Diagrams) and organizing the problem view (e.g., Input/Output Diagrams and Flow Charts).  For a more detailed description, I suggest reading, "A new American TQM" by Shiba, Graham, and Walden, Productivity Press, 1993, ISBN: 1-56327-023-3.

Though this process seems quite simple and logical, it is deceptively complex when one actually applies it.  It also works best when the development processes are already well defined and systematic data collection is already practiced.  [A rule of thumb for good metrics systems is that one collects data at one level finer granularity than the metrics require.  That allows QITs triggered by metrics to look immediately at data that provides more detail rather than having to collect it.] If you run into trouble applying this process, try time-boxing each step to force progress until you gain more experience.  (Incremental process improvement is about nibbling at problems rather than solving them all at once; so it is fair to provide imperfect solutions as a "first cut".)  If that doesn't work, get help from a consultant who is a process engineer experienced in such techniques.  There is something of a Catch-22 here.  If you have difficulty executing QITs effectively, it probably means there are systemic process problems that need to be addressed.

Blog root page
previous post in category
next post in category

6.0 Defining processes

Blog root page
previous post in category
next post in category

In this post I describe a process for defining processes.  But first I need to review what a good process definition is.  First and foremost, it is simple.  A process should be defined in a few steps (usually less than eight) and each step should be a simple imperative or a single sentence.  One manages complexity by decomposing processes from a high level of abstraction by decomposing process steps into subprocesses.  However, each subprocess is a standalone process in its own right. The level of abstraction is important.  In many software shops the development process may not be defined any more finely than: (1) Design; (2) Code; (3) Test.  That is a perfectly valid process definition for such a shop.  One provides greater process definition only when consensus exists for finer detail or there is a demonstrated need to do so. At that point hierarchical decomposition through subprocesses gets to the level of detail that is deemed necessary.  (Note that not all process steps need to be decomposed to the same level; only those that need it are decomposed.)

This hierarchical structure through levels of abstraction also provides a stopping criteria for process definition.  The criterion is quite simple: stop decomposing when one cannot reach consensus on a single set of process steps at the next lower level of abstraction.  Unfortunately this is a point that is often missed during over-eager attempts to define processes.  Processes are designed to help people do their jobs.  If the people doing the work can't agree on how it should be done, you aren't helping them by trying to ram a uniform process down their throats.  So if you can't reach consensus on the steps in a process after due diligence, back off and define the process only down to the next higher level of abstraction.  (When there are problems that can only be resolved by mandating processes, one is into process improvement and there are different techniques for doing that.)

Though the simple definition of steps through hierarchical levels of abstraction is the kernel form of process definition, there is no law against providing supplementary documentation to assist those executing the process.  This sort of documentation is attached to the process definition in some fashion (a web tool is ideal for this sort of thing).   Generally this sort of supplementary documentation is provided over time by the developers themselves.  The main goal is to provide information that may be useful to posterity.  When defining the steps there is an implicit understanding of the development environment that makes such a cryptic definition useful.  Typically new hires don't have the advantage of such knowledge and supplementary documentation becomes highly valuable to them.  It is also useful for identifying alternative detailed approaches to steps that are not further subdivided.  Finally, such documentation can provide useful check-list information about prerequisites, expected outputs, domain experts available, review policies, and whatnot.

Generally processes should be defined on a grass-roots basis by the developers that actually execute them.   Initially they simply represent the way the organization actually does things.  Over time process improvement will modify the processes when solving process problems based upon solid cost/benefit analysis.  When initially recording an organization's processes, just do it as-is; don't attempt to fix them "as long as we're here".  Leave the fixing for formal process improvement techniques (see the previous post in this category). 

[Typically when an organization documents its development process the first time there will be very few of them because attempting to provide detail usually won't reach consensus.  That's because shops without existing process documentation invariably have a lot of ad hoc processes applied by individual developers in a non-repeatable manner.  Don't try to impose order until there is a demonstrated need to do so.]

The following is a technique for defining process for the first time in an organization.  This process is time-boxed in that the amount of time spent is fixed.  This is to prevent getting bogged down in nit-picking.  The team meets four times for no more than two hours each meeting.

In the first meeting the team identifies process steps.  One way to do this is via a blitz similar to an object blitz.  The team spend an initial period (15-30 minutes) throwing out candidate steps without prejudice or any attempt at consensus.  The candidates are then individually scrubbed to make sure everyone understands what is meant by the step.  Some pruning may be done for steps that will clearly not make consensus as phrased.  There are three choices for disposition: they can be rephrased to a higher level of abstraction that is acceptable by all; they can be removed altogether; or they can by put in the "parking lot" for consideration at subsequent meetings.

Note that during the blitz a "process step" is just some activity that needs to be done in the process context.  Some will be more detailed than others in a stream-of-consciouness blitz, but it is best to try to be as detailed as possible.  Subsequent processing will coalesce the details into higher levels of abstraction.  Since this process is most effective when initially documenting an organization's existing processes, the activities should be things the organization is currently doing.  That removes the temptation to get side-tracked debating the merits of process changes.

The second meeting is spent organizing the candidate process steps.  The first task is to group the candidates into logical associations by the type of activity.  That is, activities are grouped by what they do.  This will tend to group both high and low level views of the same activities.  This will often provide an initial cut at hierarchical decomposition of steps over different levels of abstraction.  A common situation is to find two or more detailed activities that are closely related.  If there is no obvious equivalent activity at a higher level of abstraction that would naturally include them, one can create the more abstract step as a "parent".

This sort of bottom-up abstraction continues until a rough hierarchy of process steps begins to form.  This sort of initial cut is best done on a white board using Post-Its for the process activities.  Once a skeleton is in place from the bottom-up aggregation, the team shifts to a top-down view (usually about half way through the meeting).  The goal is to abstract a complete set of process steps at the highest level that includes those below.  The phrasing of these steps should be scrubbed to ensure that a consensus is made.  (That may mean ignoring some of the detailed steps that went into the original bottom-up skeleton.)

If time permits, the team can try defining subprocesses for the highest level steps in exactly the same manner.  This is done one step at a time in a breadth-first manner.  So long as time remains this process continues.  At the end of the meeting the "process" consists of only those steps that have been scrubbed until consensus is achieved in the hierarchical decomposition.  Any more detailed activities that remain are simply saved for the next meeting.  They are not included in the process definition even if everyone agrees they are something that is always done.  This is necessary to ensure the team doesn't get distracted with too much detail.  (Note that switching to the top-down view ensures that the team defines the most important parts of the process first within the limited time allowed.)

The third meeting basically cleans up the process from the second meeting and takes a preliminary cut at providing supplementary documentation.  The clean up is mainly to order the steps within each process.  However, there may be logical gaps (i.e., missing steps) in the process as defined because some detailed activities were not included.  The team must fill those gaps with process steps at the appropriate level of abstraction in the hierarchy.  That is, before addressing supplementary documentation, the team must ensure that the process hierarchy defined in the previous meeting is complete.

Once there is a complete process hierarchy definition, the team identifies items that should be included in an initial cut at supplementary documentation.  Exactly what is done here will depend very much on local policies.  For example, if the policy is to define inputs and outputs for every process activity, then the team would explicitly define them.  This is where all of the candidate process steps that were not specifically included in the process definition are put.  These represent things that anyone executing the process should at least consider doing.  However, since they weren't among the formally defined process steps, they should be considered optional until they are explicitly included via process improvement or another iteration on process definition.

After the third meeting the team members usually have homework to do for the next meeting.  Basically the process description, including supplementary data, must be expressed in its final "production" form.  How much work is involved will depend upon local policies.  For example, the policy might require that each process step have a descriptive paragraph written about it that provides additional detail and writing those descriptions would have to be allocated to the team members.  Regardless of policies, some effort will have to be expended to get the material into its final form.  The period after the third meeting is also often used to distribute the process materials to a wider review audience.

The final meeting is essentially a review of the process definition in its final form to ensure everything is properly done.  Any items provided by team members between meetings are reviewed.  The last task for the final meeting to evaluate whether additional iterations are justified.  If so, the subsequent iterations would further decompose the hierarchy and the remaining candidate processes would "seed" that iteration.

Blog root page
previous post in category
next post in category

7.0 Process frameworks

Blog root page
previous post in category

Process frameworks have the most unjustified reputation of any aspect of the software development environment.  Much of the reputation is passed mostly via word of mouth among developers and that portion is steeped in ignorance.  Unfortunately some of it is perpetrated by a couple of overly zealous advocates of the agile OOP-based processes who, most charitably, don't know what they are talking about.  The result is that a number of myths have become part of the Conventional Wisdom of software development:

Myth No 1: All process frameworks are bureaucratic.  This is nonsense.  A process framework is only as bureaucratic as its implementors make it.  Alas, this myth arose because there have been a lot of very poor implementations of process frameworks.  Some of the blame for that resides in the way process frameworks are expounded.  For example, the exposition of the CMM in its main bible takes all its examples from the world of DoD mega-projects.  But the framework itself is not inherently bureaucratic despite that exposition.

Myth No. 2: Process frameworks stifle creativity.  Again, nonsense.  Process frameworks only define what one should do, not how one does it.  In addition, most process frameworks say nothing at all about the intellectual content of software.  That sort of creative intellectual contribution is left to specific A&D methodologies.

Myth No. 3: Process frameworks impose a waterfall model.  This is generally true.  However, the implication is that the waterfall is large scale (i.e., one waterfall per project).  A few antiquated process frameworks did have this weakness.  But newer process frameworks fully embrace incremental, iterative development life cycles.  So the scale of the waterfall can anything down to the code fragment level.  Even the agile, OOP-based processes embrace the waterfall for their iterations.

Myth No. 4: Process frameworks are meant for huge projects with a cast of thousands.  Much of the research that produced process frameworks was, indeed, triggered by such projects because they are very difficult to manage and historically they have had high failure rates.  However, the principles behind modern process frameworks are borrowed from the well-established field of process engineering and they can be applied independently of scale.  What differs between large and small scale implementations is the way the framework is implemented.

Myth No. 5: Process frameworks are rigid and inflexible.  Some are.  But the good process frameworks, which includes almost all of the modern ones developed after 1980, are designed to be flexible.  For example, Level 5 of the CMM is devoted exclusively to ensuring that the framework implementation is flexible and self-correcting.  [I have always found it ironic that advocates of XP tend to be among the most vocally opposed to process frameworks, yet XP is probably the most rigidly defined software development process I have ever encountered.  In my opinion the greatest weakness of XP is that it has no built-in practices for process improvement or to deal with change in the development environment.]

Now that the myths are out of the way it is time to talk about what process frameworks are really about.  A process framework is just a skeleton that organizes some set of processes in the development environment.  The main purpose of a process framework is to define what should be done to produce a quality product.  That is, the process framework defines what processes the organization needs to have in place and what basic activities those processes need to address.  The actual processes that an organization defines are the implementation of the framework.  The key phrase here is that an organization defines.  The framework does not define processes, the local organization defines the processes as it implements the framework.

This distinction between specification (by the framework) and implementation (by the local organization) is at the root of most of the misconceptions about process frameworks.  (At least in the software industry; other industries are not at all confused about the distinction.)  In fact, the best way to initially implement process frameworks is with a minimalist approach where one implements processes that are the absolute minimum necessary to meet the framework specification using a broad interpretation.  One can then employ ongoing incremental process improvement to fill the gaps when it becomes apparent that the implementation was a tad too liberal.

[Anecdotal data point.  I had the good fortune to work for many years in a small organization where everyone was a grownup.  In that environment many of the processes we implemented were based on verbal Hallway Decisions where there were no paper trails signed in blood and stored in vault under the Rockies.  Thus no one physically signed a specification cover sheet; their attendance noted in the review meeting's minutes was sufficient.  Our shop was knee-deep in well-defined processes specified by the CMM but there was very little bureaucracy.  That was demonstrated by the fact that we maintained productivity that was integer factors better than figures published by industry Players for comparable software and our defect rates were at 5-Sigma for commercial grade software.  I attribute that degree of success to the fact that we were a process-oriented shop.]

Another thing to note about process frameworks is that there are a lot of different flavors, each with different goals.  Frameworks like TQM and Baldrige were initially focused purely on process improvement techniques rather than the processes themselves and they had much broader application than just software development.  Frameworks like ISO and CMM also have much broader application than software but they focus on what processes are necessary in a given environment.  Frameworks like RUP are a hybrid of process and methodology specification.  This segues to a distinction that I find useful when contrasting process vs. methodology.

Process.  A series of activities necessary to achieve a defined goal, usually expressed as a sequence of steps.  In software development environments the activities manipulate physical artifacts like work products.

Methodology.  A process where the activities are primarily intellectual and only the final goal may be physically manifested (e.g., a stored model or code).

Generally processes in the sense above are not very creative, other than the act of designing the process itself (which is a lot like OO development because one uses abstraction to capture invariants).  The processes that a process framework specify usually just provide a step as a place holder for a methodology that is executed to produce a work product for that step.  Thus the notion of creativity is largely irrelevant to process frameworks.  One employs process frameworks to organize the many rote activities that surround the more creative aspects of software development.

Blog root page
previous post in category