November 11, 2004
4.0 Basic process improvement
Blog root page
previous post in category
next post in category
Probably the most important activity in software development is process improvement. The technologies and tools for software development change very rapidly and that means that the processes around those technologies and tools need to change as well. In addition, process improvement is absolutely crucial to defect prevention (See the category on testing and defect prevention).
Unfortunately the industry has had remarkably bad experiences with ad hoc process improvement. Process changes get lost so we repeat the mistakes of the past. Changes become obsolete so we waste effort doing irrelevant things. Processes "creep" over time because they are not properly documented and developers are not properly trained in them. Perhaps worst of all, ad hoc process changes are not properly integrated into the development environment so that they are inefficient and bureaucratic. As a result ad hoc process improvement simply doesn't work over the long term.
One problem is that software developers generally don't know very much about process engineering, which is a discipline that is comparably complex as software engineering. Many years ago in my naive youth I was a member of a team assigned the task of documenting our software development process. We struggled mightily for three weeks and came up with a 25-page opus. With 20/20 hindsight I now realize that document was complete crap. For example, the first 2 pages described the process and the remaining 23 pages described all the exceptions. That wasn't a process; it was anarchy.
For process improvement to be effective, it needs to be have several basic characteristics:
Incremental. Developer resources are scarce and developers cannot be sucked into a rabbit hole of effort trying to solve the process equivalent of World Hunger. Process improvement needs to be narrowly focused and "effort-boxed" so that it can be integrated painlessly into the developer's main job of producing software.
Disciplined. Process improvement also needs to take advantage of lessons others have learned. So the way one does process improvement should already incorporate good practices from other disciplines like process engineering. That is, the process discipline already formally incorporates proven practices learned in the past and in other contexts.
Systematic. To be efficient, one needs to go about process improvement without having to grope for solutions to problems that recur in different forms. One also needs to take good advantage of any tools that are available. Basically that means that many of the process improvement activities should be ingrained in developers as a matter of habit. Developers should be able to "walk" the path to process change in very much the same way during every increment of improvement.
Simple. The notion of KISS that is so cherished by software developers also applies to process improvement. One of the <many> problems with the process description in my example above was that it was far too complicated. Good process descriptions have less than eight steps, each defined in a single sentence. If more detail is required, individual steps are decomposed as new processes at a lower level of abstraction. And one quits decomposing process steps when one cannot reach consensus on a single set of steps for the process.
[Actually, process descriptions can have substantial supplementary information. For example, the supplementary information could include inputs (prerequisites) that must be in place to execute the process. But that material is peripheral to the actual process; it is simply optional documentation. The process steps themselves should be clear enough to execute for anyone familiar with the general process context. And any supporting documentation should be limited to what the developers themselves feel is important.]
Communicated. Obviously process changes need to be communicated to everyone affected and any new personnel need to be properly trained. In addition, the process of doing process improvement needs to be communicated in the same fashion. [Note that I avoided the term documented, which tends to conjure up images of huge piles of paper. Though rare, verbal communication can be adequate in some contexts. The point here is that the form of the communication doesn't really matter; whatever works. In addition, KISS applies to the communication media as well.]
Data collection. Good process improvement is impossible without hard data. However, it is extremely important that data collection be minimally intrusive to ensure effort is not diverted from software production and process improvement. Therefore as much care should be devoted to providing good data collection systems as is devoted to providing good process improvement. The two should be regarded as being inextricably linked.
Probably the greatest pitfall in doing process improvement is ignorance. As I indicated above, software developers rarely know anything about process engineering. (A sad commentary on the state of CS education, in my opinion.) That's why the alphabet soup of process frameworks exist: CMM, TQM, ISO, et al. Such frameworks distill the academic discipline of process engineering into standards, guidelines, and other infrastructures that support process improvement in software development.
[Unfortunately, the second greatest pitfall in process improvement is taking such frameworks too literally. For example, the CMM has a horrific reputation among software developers for bureaucracy because of a few misguided implementations. In reality the CMM just defines what one needs to do, not how to do it. But too often shops look at examples drawn from DoD projects with a cast of thousands spread over three continents and then apply them literally to a team of a dozen developers with disastrous results. But that is fodder for another suite of posts...]
In the next couple of posts I will describe some very basic processes that can be employed for systematic incremental process improvement. Practicing these two processes will go a long way towards implementing process improvement in a development environment. But in the long term one really needs a full process improvement framework.
Blog root page
previous post in category
next post in category