While it is certainly true that the tools and methodologies of the software development process have improved dramatically, it is equally true that the complex problems we are attempting to solve are growing faster than our ability to comprehend them. This is not a temporary aberration. It is axiomatic that our technologies will always lag behind our understanding. Therefore, progress cannot be made if we keep insisting on deterministic solutions. Progress will only be made when we begin to look for the deep order within chaos. That is, when we begin to recognize meaningful patterns.
The first chaos theorists, the scientists that set the discipline in motion, shared certain sensibilities. They had an eye for pattern, especially pattern at different scales at the same time. They had a taste for randomness and complexity, for jagged edges and sudden leaps .
We must learn to amplify patterns that increase the probabilities of success, and minimize (or eliminate) patterns that increase entropy. Our sensibilities must become more attuned to the descriptive, rather than the prescriptive. This is out of necessity. It is the best that we can do (for now).
I will use the “X” diagram below to illustrate why IT organizations have had such a difficult time delivering viable technology enabled solutions. IT organizations came into existence in order to provide a bridge between the business world and technologies that were beginning to have an impact on the business landscape. Initially, their role in life was to automate existing manual processes related to back-office systems (accounting, payroll, purchasing, etc.). The rectangle depicts the solution space that was IT’s purview.
As you recall from Chapter 1, in the days of IBM domination, all four aspects within the rectangle were essentially stable, certainly by today’s standards. However, with the advent of the destabilizing technologies previously mentioned, each aspect individually, and all of them taken together, began to go non-linear (i.e. they became chaotic). At the same time, IT was increasingly being asked to innovate, as well as automate.
The established order began to unravel, and it continues to reel from the onslaught of discontinuous change that wreaks havoc on their ability to cope. This technology stuff is no longer only an IT problem. It is an organizational problem. One that requires a radically different approach if we are going to leverage its benefits in a meaningful (and cost effective) manner.
We are never going to conquer complexity. All that we can do is become better acquainted with it, and develop an eye for the jagged edges and sudden leaps. We need to build into our processes an understanding that, at the inception phases of a complex system, we know very little about the end game. We have some vague ideas about the story we are creating, but only the creation process itself will reveal the true, and final, meaning.
The whole question of complexity is one that has proved surprisingly subtle when investigated by modern theorists. Complexity is not just a matter of a system having lots of parts, which are related to one another in non-simple ways. Instead, it turns out to be a special property in its own right, and it makes complex systems different in kind than simple ones, enabling them to do things, and be things we might not have expected .
It turns out that the whole is not only greater than the sum of its parts, but also quite different from it. Many of the E-Business systems that we are currently developing are more complex than anything that we have attempted previously. We must learn to eat the complexity elephant one bite at a time. The big bang theory is a recipe for disaster. Let me provide an example to help illustrate the complexity involved in such systems.