The work leading to this course was motivated by wondering why, in software engineering, there are some people who are one or two orders of magnitude more useful than most people. If this was true of bricklayers, the building industry would be very keen to find out why. The problem, of course, is that one can film a bricklayer, and later analyse what is happening at leisure. One cannot even see what great programmers do, and for some reason, they cannot explain what the difference is themselves, although most of them wish they could.

We knew that the elements of industry best practice alone are not enough. Management commitment to investment and training are not enough. Innovative Quality programmes that explicitly include holistic concepts such as Robert Pirsig's Zen and the Art of Motorcycle Maintenance, which much of the industry would consider too radical to experiment with are not enough. Years of experience are not enough, nor are years of academic study.

There seemed to be only one way to continue the investigation if an industry dedicated to objective metrics had not found the X factor: we needed to look at the subjective experience of the people concerned.

Achieving the understanding of what was happening took a long time, although the key ideas are known to most of us already. On the way, we learned a great deal about the mindset of successful programmers and were able to develop exercises that certainly helped many people.

Thus the material in this course has developed over several years, and is a mix of ideas empirically justified by experiment and later fitted into the logical picture, and material derived from the logical picture.

This course aims to address the element of experience' orjudgment' referred to almost everywhere but rarely described. Many of the topics are the kind of thing programmers discuss over a beer. Perhaps it is odd that no-one tends to ask how the issues that programmers see as most important relate to the `formal' structures of modern engineering. Here, we do just that.

We have found that once we get into the swing of this, most programmers find they have an opportunity to put issues they have wondered about for years into a clear work context, together with their colleagues. We, therefore, ask you to relax, because you are supposed to be doing this, and have an enjoyable time!
Mapping and Software Engineering

Software engineering is in a terrible pickle. The so-called `Software Crisis' was identified in 1968, but despite thirty years of effort, with hundreds of supposedly fundamental new concepts published, the general state of the industry is horrific. Projects run massively over-budget or collapse entirely in unrecoverable heaps. Estimating is a black art, and too many projects solve the customers' problems of yesterday, not today. The technical quality of most code is dreadful, leading to robustness problems in service and high maintenance costs. And yet within the industry there exist individuals and groups who enjoy staggering, repeatable successes. There are many ways of measuring the usefulness of programmers, but some are rated as over a hundred times more useful than most, by several methods of counting. If only the whole of the industry performed as well as the tiny minority of excellent workers, the economic benefits would be immense. If it were possible to write sophisticated and reliable software quickly and cheaply, the intelligence of society would increase, as everything from car sharing to realistic social security regulations became possible.

Within this model, the problem can be understood. What is presented as socially conditioned conventional thinking (called packing) is based on action. To be a good bricklayer, a packer must know what a bricklayer does. What does a programmer do? The most developed packer model of programming is the concept of the Software Factory. In this, statements of requirements from customers go in one door and are processed by workers, following procedures, written down in manuals. When the production line has done its work, programs come out of the other door. It works in car factories.

The trouble is, the analogy with a car factory is sloppy. Most of the car factory is filled with workers using machines to make cars, but around the back, there is a little office where another worker determines how to use the resources of the factory to make as many cars as possible, all alike.

The workers in a software shop are not like the factory floor workers. The shop floor workers can be replaced by robots today, but the person who uses creativity to set up the factory is still needed. The programmers are doing the same job as the office at the back of the factory, and we cannot learn anything about what happens in there by playing at car factory shop floors.

Packers who advocate uncompromising process-based Software Factories are in fact claiming to be able to implement an Artificial Intelligence that simulates a production line designer and to be able to do it by using humans pushing bits of paper around as their computer. Unfortunately, packing is just not up to the job of understanding software production and gets terribly confused. This means it says some very silly things sometimes.

To understand what programmers really do, an alternative strategy of thinking (called mapping) is necessary, because programming is essentially a process of internalizing the capabilities of the system, the nature of the problem, and the desire, and capturing the insight in a programming language. It is all about exploring the details of our desires and understanding them in such a way that we can keep track of all the complexity. Mapper problem collapse can produce beautiful, tiny, elegant programs with no room for bugs in them. Mapping can do programming, but how it does it cannot be explained in packer's, action-based language.

Packers, therefore, assert that hackers are `irresponsible' and discount their work, saying that complexity is inherently not understandable and we must develop ever more complex procedures to abdicate our responsibility to.

Fortunately, many organizations' management continues to foster reflection on grounds of personal intuition and empirical experience, without any justifications to place on action-based balance sheets. This is a difficult thing to do but is the only reason anything gets done.

It is important to recognize that mapping is not another procedural methodology to be applied to a packer mindset. It is a different way of looking at things altogether. It is necessary to convince yourself that it really is possible to take personal responsibility for an undertaking instead of abdicating in favour of a procedure.

Programming is as near to pure mapping as you can get outside your skull. This is why it is fun. It is an endless discovery, understanding, and learning.

Object Orientation (OO) and mapping have an interesting relationship. OO is often seen in very different ways by mappers and packers. The mapper's map is a kind of object model that has a rich variety of objects and associations. Mappers see OO as an elegant way to design software once they have understood the problem. Packers seem to see OO as a way of wandering around the problem domain and creating software objects, then just wiring them up as they are found. Thus OO is taken to be a procedural mechanism for getting from problem to program without the intervening understanding. If it were possible to capture absolutely every aspect of the problem domain and one did not care about efficiency, this approach might even work. But in fact, good taste is always needed in object design and categorization, because it is necessary to design software objects that have a good mapping with real-world objects but can be plugged together to construct a viable computer system. That takes understanding and to put it strictly - a mapper job. This explains the OO projects that grind to a halt with the product a tangle of real and utility objects using multiple redundant addressing schemes to communicate via Object Request Brokers, with no clear conceptual integrity in instantiation, flattening and journaling. Packer programmers often have so little control over their objects that they lose them, and end up with memory leaks that cause the application to fail. The packer solution to this is to buy a memory leak detection tool, rather than to regain control of their objects so that everything else works properly too.

Copyright (c) Alan G Carter and Colston Sanger 1997