Because it helps explain the variability and randomness of non-linear systems, chaos theory can be used to understand many of the issues seen in software development, explain many past failures to effectively change organizations and processes, and imply areas for further research and action. It explains why reducing process variation at SEI level 4 isn't enough to ensure process optimization at SEI level 5. Since in non-linear systems even the slightest variation can have dramatic results, it is important to understand the impact and organization of any underlying factors that affect process and product outcomes.
Side-by-side with chaos theory, mathematicians discuss fractals; those colorful self-similar shapes often associated with computer graphics. The root of fractal geometry is the study of fractional dimensions (e.g. an infinite length line drawn in a finite space is more than a one-dimensional line, and less than a two-dimensional plane).
Example: How long is the Atlantic coastline of the United States? The answer varies based on the length of the measuring device used. An automobile wandering the coastal highways while keeping the coastline in sight will arrive at a different answer than the hiker who walks keeping the coastline within a few paces. The hiker determines that the coastline is quite a bit longer than the driver. A dog walking along the edge of the water would measure a longer distance still. To the ant, the coastline is many orders of magnitude longer than for the driver. The more granular the measuring device, the longer the result achieved. At the microscopic level, the coastline approaches an infinite length. It becomes the infinite line in finite space: a fractal.
The idea of self-similarity in fractals comes from the fact that the driver, hiker, dog, and ant would observe very similar geometry. Series of relatively straight stretches would be punctuated by rough edged dips and curves, often folding back on themselves. This geometry would remain consistent whether the point of view was the driver (a very large scale view) or the ant (a very small scale view). Self-similarity in fractals raises questions about what can and can't be objectively measured.
Regardless of the level of granularity in any estimate, each will include starts and stops, interruptions, and side-tracked activities; they're self-similar. Because measuring distances involves selecting measuring units and granularity, whether the length of a coastline or the duration of an activity, continuing efforts to decompose a software engineering effort into ever finer discrete components beyond CMM level 3 may actually be contrary to the objectives of the CMM.
Chaos theory looks at entire systems, not discrete components whose individual roles aren't necessarily understood. It asks for the underlying order that remains hidden among the visible self-similar chaotic results. The underlying order in a potentially chaotic system is known as the system's strange attractor.
A clear sense of purpose and delineated standards can form the strange attractor for a software engineering organization that has matured as far as CMM level 4. Remaining fluctuations and variations at the personal or activity level smooth out, or cohere, over time into definitive and predictable forms. Chaos theory tells us that, given a chance, just a few underlying concepts or themes (i.e. Deming's profound knowledge) can be used to explain the wide variety of chaotic results often perceived in organizations. The result is that an effective organization at CMM level 5 may actually require fewer standards and procedures to be in place than one at CMM level 4.
The challenge is to quantify those central themes using SPC at CMM level 4. Information technology management must allow the underlying order of each organization to emerge naturally through the day-to-day activity of its components. An inverse relationship is likely to exist between the clarity of an organization’s mission and purpose and its dependence upon formal rules and procedures. The more-is-better approach to moving from CMM level 1 to CMM level 3 gives way to a less-is-better approach to optimization by CMM level 5. Processes that are likely to be described by robustness entering CMM level 4 evolve toward characteristics such as elegance and simplicity by CMM level 5.
Linear systems can be described as being in one of three possible states: static, convergent, or oscillatory. Real-world processes are best described using the oscillatory state model. They tend to exhibit variability along virtually any dimension studied. This variability is demonstrated through the creation of control charts that plot inherent, or systemic, variability against a calculated mean. In the absence of variation, the mean would represent the process in a static state. Implementation of process improvement efforts, geared toward reducing process variation, move the process from the initial oscillatory state into a state of convergence.
As any software professional can testify, some improvement efforts not only fail to change the process state from oscillatory to convergent, they can actually worsen the behavior of the system. Changes that have worked well in the past, or in different organizations, suddenly and unexpectedly result in degraded process performance. Often the degradation is dramatic and the result is new and overwhelming complexity.
This unexpected result occurs because software engineering processes are not linear systems, they are non-linear. Non-linear systems exhibit a discontinuity between inputs and outputs; changing inputs a little can have a variety of impacts upon the output, ranging from nothing to enormous. Non-linear systems can be described by four possible states: static, convergent, oscillatory, and chaotic.
The increased complexity that occurs as the result of some process changes is unexpected because the system being changed is presumed to be linear. Under a non-linear model, an oscillatory system described by a process control chart is always balanced between moving toward convergence, or into chaos. Most real-life systems are non-linear, and non-linear systems can be chaotic. Chaos theory helps to explain the behavior of non-linear systems by describing an inherent order that underlies the surface complexity.
Statistical Process Control is an attempt to move a system from an oscillatory state to a convergent state without allowing it to slip into a chaotic state. The continuous movement of a process through convergent states to a static state at some optimum level is an implicit goal of SPC. However, the goal of dropping process variability to zero is elusive. SPC exhibits fractal geometry over time. Regardless of the level of improvement seen in a process, a control chart describing that process always looks the same. Over time, the only part of a control chart that changes is the scale of variability. Variability, once measured as days or centimeters, later is measured in hours or millimeters. The scale reduces, but the inherent variability exhibits the same repeating fractal patterns over time.
By the time an organization has matured its software processes further, however, the process variances attributable to inconsistency and noise have been largely eliminated (i.e. the low hanging fruit has already been picked). Remaining variation is attributable to finer scale measures among interdependent variables, the changing of which could drive the system into chaos. As a result, regression analysis has become the dominant SPC statistical technique, over control charts, by the time the organization emerges from CMM level 4 having attained a CMM level 5 process maturity.
Extending SPC to include various models and ideas from chaos theory provides benefit by allowing insights gained in the study of chaos to be applied almost immediately to process improvement efforts. This immediate use of new information is wholly consistent with CMM level 4 and statistical control charts. Institutionalizing this practice, along with the introduction of regression analysis, is necessary to effectively move the organization to CMM level 5.
If, in the terminology of chaos theory, vision and mission are the strange attractor for organizations that are otherwise in a constant state of chaos, then clarifying and communicating the organization and project's sense of mission can provide the few simple rules needed by software professionals to self-manage their own activities. Chaos theory predicts that the results, while not explicitly controlled, will remain reasonably close to some desired set of measures.
"If we can trust the workings of chaos, we will see that the dominant shape of our organizations can be maintained if we retain clarity about the purpose and direction of the organization. If we succeed in maintaining focus, rather than hands-on control, we also create the flexibility and responsiveness that every organization craves. What leaders are called upon to do in a chaotic world is to shape their organizations through concepts, not through elaborate rules or structures."
Recent research into the use of chaos theory offers guidance in how SPC can be integrated into CMM level 4 practices.
To be of use to the software professional, chaos theory must be more than academic. It must illuminate unexplained historical observations and it must predict behaviors that will help improve process performance in the future. Chaos theory offers a framework for conducting SPC that goes beyond traditional practice.
More research is required to formalize new and extended software quality practices. Enough is evident now, however, for the CMM to immediately benefit from this emerging field of study.