Erik Naggum's Ideas and Principles

Size and Complexity

Most systems worth working with vary from huge to gargantuan in size. What makes them worth working with is that this size can be mastered. What makes it possible to master size is human intelligence applied to reduce the complexity caused by lack of understanding. This is true for almost anything in this world except software.

Computers have generally been puny in capacity, but have still solved huge problems. Computer and software engineers have been able to make do with underpowered systems and have managed to squeeze the last drop of usefulness out of machines. This is a very valuable ability, but its main problem is that it works well only as long as the size of the system they squeeze performance out of is so small that they can keep all of it in their memory, i.e., it does not scale very well. When this methodology is applied to larger systems, the usual method of dealing with size and complexity is to partition systems into "modules" that are believed to reduce the complexity faster than they reduce the size of each module, evidence to the contrary notwithstanding. I think this is the reason for our failure to handle large systems in computers: size and complexity in information is not modularizable.

Reduction in complexity in information may be obtained when two or more ideas that are combined to produce a desired result are shown to have a commonality that makes each an application of a unifying principle. Science is rife with such developments -- indeed, it is this that is science. In brief, one may say that scientists take things apart, while engineers put things together. The effect of having engineers work on problems of science should be obvious, and vice versa.

For decades, we have been able to put things together, and Software Engineers have succeeded in solving tremendous problems. Since the engineers have been so succesful, and the Computer Scientists have been unable to reduce the insidious complexity of software systems, it has become customary in some quarters to ridicule the abstractions that Computer Scientists work with, while seemingly small problems are solved with incredible amounts of code and and unbelievable hardware capacity.

Underneath all the complexity in these huge, ever-expanding systems, there must be many unifying principles struggling to get out. If we hope to be able to cope with the size, we must deal with the complexity the right way: by discovering those unifying principles.