November 11, 2004
6.0 Defining processes
In this post I describe a process for defining processes. But first I need to review what a good process definition is. First and foremost, it is simple. A process should be defined in a few steps (usually less than eight) and each step should be a simple imperative or a single sentence. One manages complexity by decomposing processes from a high level of abstraction by decomposing process steps into subprocesses. However, each subprocess is a standalone process in its own right. The level of abstraction is important. In many software shops the development process may not be defined any more finely than: (1) Design; (2) Code; (3) Test. That is a perfectly valid process definition for such a shop. One provides greater process definition only when consensus exists for finer detail or there is a demonstrated need to do so. At that point hierarchical decomposition through subprocesses gets to the level of detail that is deemed necessary. (Note that not all process steps need to be decomposed to the same level; only those that need it are decomposed.)
This hierarchical structure through levels of abstraction also provides a stopping criteria for process definition. The criterion is quite simple: stop decomposing when one cannot reach consensus on a single set of process steps at the next lower level of abstraction. Unfortunately this is a point that is often missed during over-eager attempts to define processes. Processes are designed to help people do their jobs. If the people doing the work can't agree on how it should be done, you aren't helping them by trying to ram a uniform process down their throats. So if you can't reach consensus on the steps in a process after due diligence, back off and define the process only down to the next higher level of abstraction. (When there are problems that can only be resolved by mandating processes, one is into process improvement and there are different techniques for doing that.)
Though the simple definition of steps through hierarchical levels of abstraction is the kernel form of process definition, there is no law against providing supplementary documentation to assist those executing the process. This sort of documentation is attached to the process definition in some fashion (a web tool is ideal for this sort of thing). Generally this sort of supplementary documentation is provided over time by the developers themselves. The main goal is to provide information that may be useful to posterity. When defining the steps there is an implicit understanding of the development environment that makes such a cryptic definition useful. Typically new hires don't have the advantage of such knowledge and supplementary documentation becomes highly valuable to them. It is also useful for identifying alternative detailed approaches to steps that are not further subdivided. Finally, such documentation can provide useful check-list information about prerequisites, expected outputs, domain experts available, review policies, and whatnot.
Generally processes should be defined on a grass-roots basis by the developers that actually execute them. Initially they simply represent the way the organization actually does things. Over time process improvement will modify the processes when solving process problems based upon solid cost/benefit analysis. When initially recording an organization's processes, just do it as-is; don't attempt to fix them "as long as we're here". Leave the fixing for formal process improvement techniques (see the previous post in this category).
[Typically when an organization documents its development process the first time there will be very few of them because attempting to provide detail usually won't reach consensus. That's because shops without existing process documentation invariably have a lot of ad hoc processes applied by individual developers in a non-repeatable manner. Don't try to impose order until there is a demonstrated need to do so.]
The following is a technique for defining process for the first time in an organization. This process is time-boxed in that the amount of time spent is fixed. This is to prevent getting bogged down in nit-picking. The team meets four times for no more than two hours each meeting.
In the first meeting the team identifies process steps. One way to do this is via a blitz similar to an object blitz. The team spend an initial period (15-30 minutes) throwing out candidate steps without prejudice or any attempt at consensus. The candidates are then individually scrubbed to make sure everyone understands what is meant by the step. Some pruning may be done for steps that will clearly not make consensus as phrased. There are three choices for disposition: they can be rephrased to a higher level of abstraction that is acceptable by all; they can be removed altogether; or they can by put in the "parking lot" for consideration at subsequent meetings.
Note that during the blitz a "process step" is just some activity that needs to be done in the process context. Some will be more detailed than others in a stream-of-consciouness blitz, but it is best to try to be as detailed as possible. Subsequent processing will coalesce the details into higher levels of abstraction. Since this process is most effective when initially documenting an organization's existing processes, the activities should be things the organization is currently doing. That removes the temptation to get side-tracked debating the merits of process changes.
The second meeting is spent organizing the candidate process steps. The first task is to group the candidates into logical associations by the type of activity. That is, activities are grouped by what they do. This will tend to group both high and low level views of the same activities. This will often provide an initial cut at hierarchical decomposition of steps over different levels of abstraction. A common situation is to find two or more detailed activities that are closely related. If there is no obvious equivalent activity at a higher level of abstraction that would naturally include them, one can create the more abstract step as a "parent".
This sort of bottom-up abstraction continues until a rough hierarchy of process steps begins to form. This sort of initial cut is best done on a white board using Post-Its for the process activities. Once a skeleton is in place from the bottom-up aggregation, the team shifts to a top-down view (usually about half way through the meeting). The goal is to abstract a complete set of process steps at the highest level that includes those below. The phrasing of these steps should be scrubbed to ensure that a consensus is made. (That may mean ignoring some of the detailed steps that went into the original bottom-up skeleton.)
If time permits, the team can try defining subprocesses for the highest level steps in exactly the same manner. This is done one step at a time in a breadth-first manner. So long as time remains this process continues. At the end of the meeting the "process" consists of only those steps that have been scrubbed until consensus is achieved in the hierarchical decomposition. Any more detailed activities that remain are simply saved for the next meeting. They are not included in the process definition even if everyone agrees they are something that is always done. This is necessary to ensure the team doesn't get distracted with too much detail. (Note that switching to the top-down view ensures that the team defines the most important parts of the process first within the limited time allowed.)
The third meeting basically cleans up the process from the second meeting and takes a preliminary cut at providing supplementary documentation. The clean up is mainly to order the steps within each process. However, there may be logical gaps (i.e., missing steps) in the process as defined because some detailed activities were not included. The team must fill those gaps with process steps at the appropriate level of abstraction in the hierarchy. That is, before addressing supplementary documentation, the team must ensure that the process hierarchy defined in the previous meeting is complete.
Once there is a complete process hierarchy definition, the team identifies items that should be included in an initial cut at supplementary documentation. Exactly what is done here will depend very much on local policies. For example, if the policy is to define inputs and outputs for every process activity, then the team would explicitly define them. This is where all of the candidate process steps that were not specifically included in the process definition are put. These represent things that anyone executing the process should at least consider doing. However, since they weren't among the formally defined process steps, they should be considered optional until they are explicitly included via process improvement or another iteration on process definition.
After the third meeting the team members usually have homework to do for the next meeting. Basically the process description, including supplementary data, must be expressed in its final "production" form. How much work is involved will depend upon local policies. For example, the policy might require that each process step have a descriptive paragraph written about it that provides additional detail and writing those descriptions would have to be allocated to the team members. Regardless of policies, some effort will have to be expended to get the material into its final form. The period after the third meeting is also often used to distribute the process materials to a wider review audience.
The final meeting is essentially a review of the process definition in its final form to ensure everything is properly done. Any items provided by team members between meetings are reviewed. The last task for the final meeting to evaluate whether additional iterations are justified. If so, the subsequent iterations would further decompose the hierarchy and the remaining candidate processes would "seed" that iteration.