Big Oil Company could lose an estimated $100 Million USD on a project and have it go virtually unnoticed outside of the organization. Big Natural Gas Company’s fate would be quite different. It was the early 1990’s, and client-server technology was all the rage. PowerSoft had developed PowerBuilder, a product that enabled the IT masses to finally break into the MS Windows development universe. Prior to PowerBuilder, only the initiated priesthood was capable of making sense of the Windows ‘C’ based API hieroglyphics. The combination of PowerBuilder on a Window’s 3.1 client, and Sybase running on a Unix server, created the client-server platform for mainstream IT.
The young Turks in senior IT management positions at Big Natural Gas Company were quick to seize the opportunity. They managed to convince the CEO that competitive advantage could be secured by becoming an early-adopter of the client-server craze. They had plenty of help from a local emerging PowerBuilder-inspired consulting company. It was, to say the least, an ambitious plan. All of Big Natural Gas Company’s strategic enterprise systems (pipeline scheduling, nominations, and gas accounting, to name a few) were to be converted to the new platform. This new platform would allow the company to integrate its customers into its operations in an unprecedented way. (Sound familiar?)
Anyone familiar with pipeline systems will attest to their complexity. They support a large customer base. They maintain complex contractual relationships between the pipeline and its customers. They must process a significant number of transactions that occur on a daily basis, and within rigorous time constraints. The mathematical complexities of systems such as pipeline scheduling are not like those of your run-of-the-mill back-office systems. It is safe to assume that hundreds of man-years had been spent developing the MVS mainframe based systems. The young Turks did not take a toe-in-the-water approach. Knowingly or not, they had convinced the business executives to bet the company.
In many respects, PowerBuilder was an ugly (and in its early incarnations, barely usable) product. It was slow (interpreted) and had a fairly large footprint. What made PowerBuilder a compelling technology was an internal object called the Data Window. The Data Window allowed PowerBuilder clients to interact with SQL databases in powerful ways. It provided both developers and sophisticated end-users an elegant window to their data. The combination of SQL and PowerBuilder provided a viable (if somewhat slow) platform for departmental client-server computing.
The problem was that Big Natural Gas Company was not porting departmental systems to the new client-server platform. They were porting their strategic enterprise systems. These mainframe-based systems supported hundreds of users and a large number of transactions. The combination of MVS, COBOL and DB2 (although not cool by today’s standards) provided a solid architectural foundation that was field-proven.
The fact that client-server was an immature platform for these kinds of systems is an understatement. If there were voices of caution that dared to speak, they were drowned out by the certainty of the young Turks, and by corporate America’s love affair with all things new. What follows is the story of an industrial giant’s demise.
I was hired to work on the C++ based Unix backend. PowerBuilder was used for all the client-side GUI work, and C++ was used for the heavy lifting on the server. The lion’s share of the staff (upwards of 120 employees and contractors at the height of development) worked on the client, even though the lion’s share of complexity existed on the server. In addition to the nominations and scheduling systems, the server hosted a number of infrastructure related systems, such as batch job scheduling.
There was some serious top gun talent hired to work on the server-based systems (Billy the Kid, Wyatt Earp, and Doc Holliday, to name a few). Even though the gunslingers had, more or less, delivered on their end of the bargain, the whole system began to implode after I had been on the job about 6 months. About this time, bad rumors started circulating. Big Natural Gas Company had not been able to accurately account for pipeline transactions for over a year, and customers were having difficulty entering transactions over dial-up lines.
Executive management was becoming impatient with the burn rate and the endless unfulfilled promises. Employees were mandated to work 60-70 hour weeks, and were obviously showing signs of burnout and fatigue. System performance was going from bad to worse. What was the executive team’s brilliant response? Spin out a software company to recoup the investment by selling the system to competitors. Hey, if your systems are fucked up, you might as well level the playing field by fucking up the systems of your competitors, right?
Needless to say, the rings of software hell just got deeper and deeper. Every customer that it managed to land sued the ill-fated spin-off. The young Turks escaped to the west coast, where they found work in unrelated industries. Billy the Kid founded a very successful software company. Wyatt Earp disappeared into the technology underworld. And as for me, I am just hoping I can finish writing this book before tuberculosis, or some self-inflicted wound manages to kill me. Big Natural Gas Company got out of Dodge as well; a competitor bought it at a fire sale price (so I hear). The estimated back-of-the-envelope loss related to this endeavor: $500 million USD (easily).
So what’s the point? The point is not to assign blame, although rumor has it that the young Turks are back in town and are up to their old tricks again. Caveat Emptor! The point is that when patterns begin to emerge, there is an opportunity to make sense out of chaos. The chaos becomes somewhat predictable, paradoxically. There are lessons to be learned and insights to be gained that can improve the probabilities of success.
The rest of this book is an attempt to highlight recurring patterns. Patterns emerge in all naturally occurring complex systems. They provide the keys to a better understanding of their inner workings. The same holds true in the development of complex software systems.