Law of Tangental Complexity

Abstract

Over the years, working on large scale software systems, I have observed that many production systems tend to approach a similar level of complexity — largely independent of the technology, architecture & programming language used. It is as if complexity “fills up” to a glass ceiling which then becomes difficult to transcend. The theory is that this glass ceiling is  a sort of  “cognitive horizon,” a psychological limit on what a humans can comprehend.

I would like to propose name for this dynamic: Law of Tangental Complexity. The tenet of this law may be summarised as follows: On any sufficiently large system, complexity will tend to “fill up” to a cognitive horizon and thereafter behave as a tangent to that horizon. The idiom of a horizon represents similarity to our natural field of view. The idiom of a tangent implies that complexity may move beyond our field of view by adding to it in a lateral fashion, i.e. by simply adding breadth of functionality rather than necessarily adding increasingly complicated functionality. The latter does not seem to happen once people have difficulty comprehending the complexities of a system. Breadth of functionality, however, continues to be added. The graphic below illustrates the process.

1) At the beginning of a project interfaces are clean, dependencies are few and the code base is small. Complexity is low.

Cognitive Horizon - Beginning of Project

Cognitive Horizon – Beginning of Project

2) The project is a success. More functionality, inter-dependencies and layers of abstraction are added. But complexity is still manageable, so continues to increase.  It is now approaching the cognitive horizon.

Cognitive Horizon - Project is Successful

Cognitive Horizon – Project is Successful

3) Commercial pressures prompt more functionality to be added. Complexity now passes the cognitive horizon. Individual team members understand individual aspects of the system. Technical leads understand the whole but not all parts in full detail. Staff turn-over and attrition lead to loss of knowledge. New hires are trained. Full knowledge of the system becomes “patchy.”

Cognitive Horizon Exceeded

Cognitive Horizon Exceeded

4) Commercial pressures persist. Team members begin to get a sense of complexity and increasingly refrain from adding more complicated features, but continue to add breadth. Because dependencies in the system are less well understood now, technical debt results.

Cognitive Horizon - Technical Debt

Cognitive Horizon – Technical Debt

Elaboration

In any large scale system layers of abstraction and dependencies tend to be built-up to accommodate new and modified functionality until the team as a whole begins to lose view of the complexity of the system. It is suggested that the factors at play are limits imposed by Cognitive Load Theory, as well as  team dynamics, not primarily by technology. For these limits to be reached, the system must typically first be successful. Any successful tool, software or otherwise, has a tendency to be progressively put to uses unindented by its original designers — until it breaks. The same dynamic bears out in large software systems. Unlike the tools of craftsmen which might snap or break, because material fatigue sets in or for similar reasons, software does not directly “break.”  Software systems, by contrast, will become unmanageable or exhibit a high number of feature defects.

Iteration

Once technical debt has been realised, successful project managers will direct their teams to refactor to make the system manageable once more. Another typical response is to hire more capable engineers.  Initially at least this will appear to resolve the issue as a more capable team with better skills will have greater cognitive abilities and this expands the cognitive horizon. Architecture is revised, better solutions are deployed. More and enhanced functionality can be accommodated. The project team is lauded. The system becomes more successful as more customers are aquired. Within a period of time, the system returns to being unmanageable as  the system complexity  “fills up” to the new cognitive horizon.

The problem has actually been compounded as the system is “trapped in a pincer movement” between a now much wider cognitive horizon and the  law of diminishing returns : Recruiting more exceptionally talented engineers who can cope with the cognitive horizon of the system proves less fruitful upon later iterations of this cycle. Lesser talent has no hope of approaching the complexities of the system at all. At this stage, a replacement for the system is commissioned and the life cycle of the original system ends. Usually, at this stage, technological reasons will be advanced as to why the original system is incapable of meeting new demands.  One example the author has witnessed is the migration of a large scale system from Java to C++:  Among other considerations, the Java garbage collection had introduced unacceptable pauses given evolving latency requirements. Other solutions might have been conceivable, including a low-pause or no-pause collector. But in truth the complexities of the “memory management” abstraction was beyond the horizon of the team.

Traditional Mitigation Strategies

  • One mitigation strategy is to limit the complexity of the system and follow the Unix philosophy “Rule of Parsimony”. This is a challenge for two reasons: 1) Commercial pressures will tend to ensure that the nay-sayers are ignored and those who suggest additional featured  can be accommodated are listened to. 2) Quantifying complexity in any system is inherently difficult — no consensus exists on how this should be done.
  • A second strategy involves refactoring a complex system into two or more smaller subsystems with lesser complexity each. In practice this too may prove a challenge as inter-dependencies between the newly created subsystems may prompt the complexity of the whole to be greater than the sum of the complexities of the parts.

Proposed Mitigation Strategies

  • One cannot change what one cannot measure. Given that software engineers work with a largely fixed toolset (IDEs, compilers, profilers, etc.), one strategy might involve equipping compilers with a metric for complexity.  Just because no consensus exists is not a reason not to make an attempt at a metric. There are many measures of complexity, ranging from space, time, computational workflow and many more are conceivable. One suggested approach is to incorporate results of escape and the related shape analysis of whole program optimizing compilers. Classical examples of compilers incorporating such analysis include the Moscow ML and Stalin Scheme compilers. These don’t emit a complexity metric but could be augmented to do so. To be useful, more mainstream compilers such as GCC would also need to be modified. Equally profilers might emit a metric on number of threads in concurrent programs and the ways in which synchronisation objects connect the workflow of such threads.
  • If layering abstractions adds complexity, then having malleable abstraction becomes paramount in managing complexity. Malleable abstraction is a synonym for meta-programming. Yet few programming languages are optimized for meta-programming.  Indeed, the orthogonality  of features in most programming languages collapses when meta programming is introduced. This has generally made meta-programming the exclusive playground of elite programmers and has given the technique a reputation as being intractable. One classic example of the breakdown in feature orthogonality in the face of meta-programming techniques is C++. Indeed there is but one family of languages that has stood the test of time here: the Lisp family of languages. Lisp is optimized for macro or meta programming in a way that no other programming language is. Traditionally this has forced the programmer to work in an abstract syntax parse tree, the core data structure of a compiler, adding to the unpopularity of the language. More recent advances have removed this constraint — see the Julia programming language for further reading on the subject.

Comment is invited.

Advertisements

18 responses to “Law of Tangental Complexity

  1. Ernst van Waning

    The advantages of Lisp you mention are not exclusively attributable to
    the fact that Lispers think in terms of parse trees — maybe they
    don’t. There are large code bases in Lisp developed over many years
    that are continually extended. Sometimes these code bases have been
    radically improved in a matter of weeks by people new to the project.

    It may not be crystal clear why this is so. One might guess this very
    attractive property of Lisp (safe investments) has everything to do
    with its dynamic aspects: many, if not all concepts in Lisp can be
    adapted. This makes it possible for people to fully concentrate on
    the problem at hand and not be overly distracted by assumptions
    unknown to them elsewhere in the code.

  2. Interesting article.

    I’ve been dealing with / thinking about this problem for over 3 decades.

    My conclusions/comments are:

    – Use compilable diagrams wherever possible, e.g. Flow-Based Programming (FBP) http://www.jpaulmorrison.com/fbp/, Drakon http://drakon-editor.sourceforge.net/, roll your own https://github.com/guitarvydas/vsh .

    – Eschew call-return (implicit synchronization) at the architectural level.

    – Eschew abstraction, be specific, solve one problem at a time in the most obvious manner.

    – Listen to Alistair Cockburn’s ideas http://www.infoq.com/presentations/cockburn-bury-not-praise-agile.

    – CL has no syntax. You program in it using concrete parse trees, hence, it is a wonderful fit for meta-programming. CL’s reader macros are under-utilized, one could graft numerous syntaxes onto CL if one wished https://github.com/guitarvydas/paraphrase .

    – See also Rebol and Red

    – See also TXL http://www.txl.ca/

    Paul Tarvydas
    (first-name dot last-name at gmail.com)

  3. First, Kudos … it’s an awesome article. It reminds me of entropy in physical engineering.

    Our board consists of one software engineer (myself), a person with a master’s degree in communication and another with a master’s degree in psychology with an emphasis in cognitive neuroscience.

    We have observed many of the phenomena you observed in your article. Our psychology person observed: “Every time you need to connect to another system, everything slows down and we can’t predict a schedule”.

    In your observation, “interfaces are clean, dependencies are few”, I think you have found the key. It’s controlling the number of dependencies and interfaces within each sub system.

    I can’t imagine a generalized technical tool to measure the complexity of interfaces and dependencies. I think the solution to identifying the “Cognitive Horizon” may be human, not technical.

    We’ve basically gone to a modular approach to manage complexity (similar to the Unix approach you mentioned), defining our software by interfaces and modules. I explain to our board what the software does and how at a high level. Our board member with the degree in psychology is not really listening to what it does, as much as she is listening for how well I understand what I’m explaining. She let’s me know when I’m starting to lose my cognitive grip, before I actually lose that grip.

    So, before we actually reach what you call the Cognitive Horizon we determine if it’s time to refactor and re-modularize.

    So, rather than changing to new approach, we are focusing on determining ways to identifying how when to use traditional approaches more effectively.

    • P.S. Our CEO (the guy with the Master’s in Communication) really liked the the term ” cognitive horizon” as a description of what we’ve experienced.

    • The concept of using psychology to identify the “Cognitive Horizon” is a nice case of lateral thinking. I like it. As an engineer, however, I also like metrics. I wonder if the two can be combined. For example, human behaviour might be a proxy indicator. Given that the “Cognitive Horizon” is both influenced by human and technological factors, i.e. the human ability to grasp complexity and the complexity that is present in a project framed by technology, might we collect performance metrics on people to draw conclusions about the “Cognitive Horizon?” What might those metrics be to rule out other factors? This line of questioning goes directly at the heart of the comment made by Paddy3118 “Where is the science?” There are two ways of arriving at the science: 1) Measure complexity and its impact on projects. 2) Measure “the psychology” as a proxy indicator. Given that neither is likely to support a single unified metric, the proxy approach might be more prone to overfitting, yet potentially more accessible.

  4. Where is the science to back up this “law”?

    • Thank you indeed for this valid criticism. In the spirit of apologetics, I think I was borrowing on Parkinsons Law which describes itself as a “adage”
      which vaguely translates as a “memorable saying which holds some important fact of experience.” As the abstract says, I have observed a dynamic and would like to propose a name. Calling it a “law” is admittedly “reaching” — on purpose. The goal is to stir controversy & draw in comments and other people’s experience. For this to have grounding in hard scientific proof, a whole different standard, one would need access to statistically representative projects metrics of a statistically representative number of organisations. The metrics would have to be of sufficient quality to be conclusive, rule out other factors, etc. Most projects don’t collect this information; most organisations would not make it available if it was collected. And so, like Parkinson’s Law, my experience will at best become an “adage,” but perhaps one shared by others. If it resonates with people, it may inspire the collection of metrics to guide projects, even if those won’t be published…

  5. Filip De Fraye

    First of all bravo, great article and well written!

    I have been thinking about complexity for some time now on different architectures. There has always been some limit I have recognized as being there and that apparently cannot be passed. This is the first time I have actually read about it in a meaningful way. It really helps me understand it from a different perspective. Thank you!

    I try to use a measure that helps me understand whether or not I’m going into a direction where the system is going to become unmanageable. It’s a very crude measure hence I’m not really sure it’s a good one.

    When you drop a set of matches, most people can – without counting – easily identify up to 6 or 7 matches, without actually counting them. Once you start dropping more, it becomes difficult and people start to actually count the number matches. There are exceptions to this rule, but that’s what they are, exceptions.

    I think the ability to understand a system somehow works in the same way. Understanding a system that only has roughly 7 components is very straightforward. For arguments sake I am using the number seven. I think it’s different for different people. These components can be high level. Each component can then be subdivided into several more components which again can easily understood if their number doesn’t surpass roughly 7. I am guessing interdependencies work the same way. If there are more than 7 “connection points” to other componenets people have to start to think about the connectivity instead of immediately “seeing” it.

    Of course, 7 is just a first limit. We should be able to use a system where we might have to think about it first. But that I think depends on what you wish to use it for. If you have to carry the system on your person and you should be able to use it immediately without having to think about it, then around 7 should be the number. Practically speaking, this might be a phone with 7 capabilities on the home screen.

    Much like David above, when I run into a system where there are complaints about complexity I try to “lower the number”, by refactoring or remodularizing. This may add a level into the system. But when the first level is too large (let’s say 20 or something), adding a level may actually start simplify it. Of course, if you’re adding more than 20 levels then you haven’t really solved the complexity issue.

    Hence I try to use some kind of gut feeling with this number. I know it’s very crude, but it helps me design systems that are – hopefully – still understandable.

    • Thank you for your feedback and I think you are onto something with the number 7 though that may be yet too large a number. I’ve always found it interesting that acronyms tend to be made of either 3 or 4 letters. 5 or more is rarely seen. This tendency may relate to what is called Working Memory, a form of short term memory that denotes what we can focus on simultaneously. We can recall 6 or 7 digit phone numbers but tend to imprint them in our short term memory by repetition or breaking them down into 3-4 digit groups. So our mind wants smaller groups than 7 even. There is an article here that discusses this http://m.livescience.com/2493-mind-limit-4.html

    • Another consideration regarding working memory is how it composes across a team. If one team member’s working memory tends to a limit of 4, then what is the team’s collective working memory if there are 5 team members? Is it simply multiplicative and tends to a limit of 20? Is it linear ? Does it tail-off as it approaches a limit imposed by communication overheads? One “accepted” limit on the size of SCRUM teams is 7. See http://scrum.jeffsutherland.com/2003/02/scrum-keep-team-size-under-7.html The similarity in numbers may be a coincidence.

      Thank you for everybody’s feedback!

  6. Valerii Potokov

    Well written!

    I tend to see the Complexity and the Сomplicacy as very different terms, but sometimes we use them “interchangeably”. Each object in Universe is very complex. We just do not go or, more often, not able to go deeper a certain border. Typically even do not need if we able to build our comprehension based on more “affordable” details.

    All kind of “natural” things like “the Honeybee”, “the Flower”, everything from, lets say, the wildlife are so cool, because they have millions years of “refactoring” process behind.

    We have no such time working even with Agila and Scrum methodologies. Our timeline if more tough 🙂

    But the idea for a necessity to measure a level of complexity in some quantitative form is really good one.

  7. To use a metaphor from architecture – Bauhaus not Victorian.

    A lot people actually prefer Victorian. Even in places where only Bauhaus can survive. They must be resisted.

    Personally I favor refactoring unless you can afford to do the whole damn thing all over again from scratch.

  8. Valerii Potokov

    I mention refactoring not because it’s a way to go, but just from a viewpoint that it works “automatically” in “the nature”. That’s why it’s hard to compete with “a perfection and optimality” of all natural things. They are complex, but not “unnecessary overcomplicated”. Its next level of functionality is built on its own “highly optimized” preceding base. Our (humans) intention to “achieve a goal in a shortest time” leaves no room comparable with an optimality and beauty of the natural things.

    The below point from the proposed “Law of Tangental Complexity” is the perfectly written summary of the above statement.

    “The idiom of a tangent implies that complexity may move beyond our field of view by adding to it in a lateral fashion, i.e. by simply adding breadth of functionality rather than necessarily adding increasingly complicated functionality.”

  9. Interesting thoughts. I’d certainly like to see it put on a scientific basis – and that science won’t be (solely) Software Engineering. SE pretends to grasp this stuff but I think some other disciplines like organisational psychology need to be in the mix.

    This is related, too, to something I’ve been musing for a few years: that these issues don’t just apply to projects/systems but to a much bigger ecosystem. One of the reasons that cybersecurity – my present bread and butter – is wildly out of control (and scaring the people who know what they’re talking about) is that the complexity of the internet ecosystem vastly outstrips your cognitive horizon. There are quite a few self-styled cybersecurity experts, but I genuinely think that none exist. Moreover, any hope one might have of getting a project/product back under some sort of control, following you mitigation strategies, largely fails for the domain of ‘cyberspace’. O tempora! O mores!

  10. Adding link to discussions on Google Plus:

  11. Pingback: How Do You Measure System Complexity? | CISQ - Consortium for IT Software Quality

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s