The advantages of Lisp you mention are not exclusively attributable to
the fact that Lispers think in terms of parse trees — maybe they
don’t. There are large code bases in Lisp developed over many years
that are continually extended. Sometimes these code bases have been
radically improved in a matter of weeks by people new to the project.
It may not be crystal clear why this is so. One might guess this very
attractive property of Lisp (safe investments) has everything to do
with its dynamic aspects: many, if not all concepts in Lisp can be
adapted. This makes it possible for people to fully concentrate on
the problem at hand and not be overly distracted by assumptions
unknown to them elsewhere in the code.
– CL has no syntax. You program in it using concrete parse trees, hence, it is a wonderful fit for meta-programming. CL’s reader macros are under-utilized, one could graft numerous syntaxes onto CL if one wished https://github.com/guitarvydas/paraphrase .
I am quite partial to the idea of flow based programming (FBP); it has the benefit of being compositional. This constrains complexity. Functional Reactive Programming (FRP) might serve a similar purpose. FBP and FRP are rather closely related…
(2) Yes FBP and FRP are related (esp. since I favour a reactive form of FBP), but, the greatest advantage to FBP is the ability to describe architecture using diagrams.
That’s a whole new high level language of software architecture. It augments the set of languages we already have.
At that higher level, it does make sense to count the number of boxes (components) on a diagram as a complexity metric, where a constraint is placed on the boxes themselves, e.g. everyone in the room agrees that each box will take less than 2 days to implement.
First, Kudos … it’s an awesome article. It reminds me of entropy in physical engineering.
Our board consists of one software engineer (myself), a person with a master’s degree in communication and another with a master’s degree in psychology with an emphasis in cognitive neuroscience.
We have observed many of the phenomena you observed in your article. Our psychology person observed: “Every time you need to connect to another system, everything slows down and we can’t predict a schedule”.
In your observation, “interfaces are clean, dependencies are few”, I think you have found the key. It’s controlling the number of dependencies and interfaces within each sub system.
I can’t imagine a generalized technical tool to measure the complexity of interfaces and dependencies. I think the solution to identifying the “Cognitive Horizon” may be human, not technical.
We’ve basically gone to a modular approach to manage complexity (similar to the Unix approach you mentioned), defining our software by interfaces and modules. I explain to our board what the software does and how at a high level. Our board member with the degree in psychology is not really listening to what it does, as much as she is listening for how well I understand what I’m explaining. She let’s me know when I’m starting to lose my cognitive grip, before I actually lose that grip.
So, before we actually reach what you call the Cognitive Horizon we determine if it’s time to refactor and re-modularize.
So, rather than changing to new approach, we are focusing on determining ways to identifying how when to use traditional approaches more effectively.
The concept of using psychology to identify the “Cognitive Horizon” is a nice case of lateral thinking. I like it. As an engineer, however, I also like metrics. I wonder if the two can be combined. For example, human behaviour might be a proxy indicator. Given that the “Cognitive Horizon” is both influenced by human and technological factors, i.e. the human ability to grasp complexity and the complexity that is present in a project framed by technology, might we collect performance metrics on people to draw conclusions about the “Cognitive Horizon?” What might those metrics be to rule out other factors? This line of questioning goes directly at the heart of the comment made by Paddy3118 “Where is the science?” There are two ways of arriving at the science: 1) Measure complexity and its impact on projects. 2) Measure “the psychology” as a proxy indicator. Given that neither is likely to support a single unified metric, the proxy approach might be more prone to overfitting, yet potentially more accessible.
Thank you indeed for this valid criticism. In the spirit of apologetics, I think I was borrowing on Parkinsons Law which describes itself as a “adage”
which vaguely translates as a “memorable saying which holds some important fact of experience.” As the abstract says, I have observed a dynamic and would like to propose a name. Calling it a “law” is admittedly “reaching” — on purpose. The goal is to stir controversy & draw in comments and other people’s experience. For this to have grounding in hard scientific proof, a whole different standard, one would need access to statistically representative projects metrics of a statistically representative number of organisations. The metrics would have to be of sufficient quality to be conclusive, rule out other factors, etc. Most projects don’t collect this information; most organisations would not make it available if it was collected. And so, like Parkinson’s Law, my experience will at best become an “adage,” but perhaps one shared by others. If it resonates with people, it may inspire the collection of metrics to guide projects, even if those won’t be published…
First of all bravo, great article and well written!
I have been thinking about complexity for some time now on different architectures. There has always been some limit I have recognized as being there and that apparently cannot be passed. This is the first time I have actually read about it in a meaningful way. It really helps me understand it from a different perspective. Thank you!
I try to use a measure that helps me understand whether or not I’m going into a direction where the system is going to become unmanageable. It’s a very crude measure hence I’m not really sure it’s a good one.
When you drop a set of matches, most people can – without counting – easily identify up to 6 or 7 matches, without actually counting them. Once you start dropping more, it becomes difficult and people start to actually count the number matches. There are exceptions to this rule, but that’s what they are, exceptions.
I think the ability to understand a system somehow works in the same way. Understanding a system that only has roughly 7 components is very straightforward. For arguments sake I am using the number seven. I think it’s different for different people. These components can be high level. Each component can then be subdivided into several more components which again can easily understood if their number doesn’t surpass roughly 7. I am guessing interdependencies work the same way. If there are more than 7 “connection points” to other componenets people have to start to think about the connectivity instead of immediately “seeing” it.
Of course, 7 is just a first limit. We should be able to use a system where we might have to think about it first. But that I think depends on what you wish to use it for. If you have to carry the system on your person and you should be able to use it immediately without having to think about it, then around 7 should be the number. Practically speaking, this might be a phone with 7 capabilities on the home screen.
Much like David above, when I run into a system where there are complaints about complexity I try to “lower the number”, by refactoring or remodularizing. This may add a level into the system. But when the first level is too large (let’s say 20 or something), adding a level may actually start simplify it. Of course, if you’re adding more than 20 levels then you haven’t really solved the complexity issue.
Hence I try to use some kind of gut feeling with this number. I know it’s very crude, but it helps me design systems that are – hopefully – still understandable.
Thank you for your feedback and I think you are onto something with the number 7 though that may be yet too large a number. I’ve always found it interesting that acronyms tend to be made of either 3 or 4 letters. 5 or more is rarely seen. This tendency may relate to what is called Working Memory, a form of short term memory that denotes what we can focus on simultaneously. We can recall 6 or 7 digit phone numbers but tend to imprint them in our short term memory by repetition or breaking them down into 3-4 digit groups. So our mind wants smaller groups than 7 even. There is an article here that discusses this http://m.livescience.com/2493-mind-limit-4.html
Another consideration regarding working memory is how it composes across a team. If one team member’s working memory tends to a limit of 4, then what is the team’s collective working memory if there are 5 team members? Is it simply multiplicative and tends to a limit of 20? Is it linear ? Does it tail-off as it approaches a limit imposed by communication overheads? One “accepted” limit on the size of SCRUM teams is 7. See http://scrum.jeffsutherland.com/2003/02/scrum-keep-team-size-under-7.html The similarity in numbers may be a coincidence.
I tend to see the Complexity and the Сomplicacy as very different terms, but sometimes we use them “interchangeably”. Each object in Universe is very complex. We just do not go or, more often, not able to go deeper a certain border. Typically even do not need if we able to build our comprehension based on more “affordable” details.
All kind of “natural” things like “the Honeybee”, “the Flower”, everything from, lets say, the wildlife are so cool, because they have millions years of “refactoring” process behind.
We have no such time working even with Agila and Scrum methodologies. Our timeline if more tough 🙂
But the idea for a necessity to measure a level of complexity in some quantitative form is really good one.
I mention refactoring not because it’s a way to go, but just from a viewpoint that it works “automatically” in “the nature”. That’s why it’s hard to compete with “a perfection and optimality” of all natural things. They are complex, but not “unnecessary overcomplicated”. Its next level of functionality is built on its own “highly optimized” preceding base. Our (humans) intention to “achieve a goal in a shortest time” leaves no room comparable with an optimality and beauty of the natural things.
The below point from the proposed “Law of Tangental Complexity” is the perfectly written summary of the above statement.
“The idiom of a tangent implies that complexity may move beyond our field of view by adding to it in a lateral fashion, i.e. by simply adding breadth of functionality rather than necessarily adding increasingly complicated functionality.”
Interesting thoughts. I’d certainly like to see it put on a scientific basis – and that science won’t be (solely) Software Engineering. SE pretends to grasp this stuff but I think some other disciplines like organisational psychology need to be in the mix.
This is related, too, to something I’ve been musing for a few years: that these issues don’t just apply to projects/systems but to a much bigger ecosystem. One of the reasons that cybersecurity – my present bread and butter – is wildly out of control (and scaring the people who know what they’re talking about) is that the complexity of the internet ecosystem vastly outstrips your cognitive horizon. There are quite a few self-styled cybersecurity experts, but I genuinely think that none exist. Moreover, any hope one might have of getting a project/product back under some sort of control, following you mitigation strategies, largely fails for the domain of ‘cyberspace’. O tempora! O mores!
Chris Kohlhepp's Blog 
The advantages of Lisp you mention are not exclusively attributable to
the fact that Lispers think in terms of parse trees — maybe they
don’t. There are large code bases in Lisp developed over many years
that are continually extended. Sometimes these code bases have been
radically improved in a matter of weeks by people new to the project.
It may not be crystal clear why this is so. One might guess this very
attractive property of Lisp (safe investments) has everything to do
with its dynamic aspects: many, if not all concepts in Lisp can be
adapted. This makes it possible for people to fully concentrate on
the problem at hand and not be overly distracted by assumptions
unknown to them elsewhere in the code.
Interesting article.
I’ve been dealing with / thinking about this problem for over 3 decades.
My conclusions/comments are:
– Use compilable diagrams wherever possible, e.g. Flow-Based Programming (FBP) http://www.jpaulmorrison.com/fbp/, Drakon http://drakon-editor.sourceforge.net/, roll your own https://github.com/guitarvydas/vsh .
– Eschew call-return (implicit synchronization) at the architectural level.
– Eschew abstraction, be specific, solve one problem at a time in the most obvious manner.
– Listen to Alistair Cockburn’s ideas http://www.infoq.com/presentations/cockburn-bury-not-praise-agile.
– CL has no syntax. You program in it using concrete parse trees, hence, it is a wonderful fit for meta-programming. CL’s reader macros are under-utilized, one could graft numerous syntaxes onto CL if one wished https://github.com/guitarvydas/paraphrase .
– See also Rebol and Red
– See also TXL http://www.txl.ca/
Paul Tarvydas
(first-name dot last-name at gmail.com)
I am quite partial to the idea of flow based programming (FBP); it has the benefit of being compositional. This constrains complexity. Functional Reactive Programming (FRP) might serve a similar purpose. FBP and FRP are rather closely related…
(1) Faison’s book http://www.amazon.com/Event-Based-Programming-Taking-Events-Limit/dp/1590596439/ref=sr_1_1?ie=UTF8&qid=1390688510&sr=8-1&keywords=event-based+programming has a good opening chapter on coupling.
(2) Yes FBP and FRP are related (esp. since I favour a reactive form of FBP), but, the greatest advantage to FBP is the ability to describe architecture using diagrams.
That’s a whole new high level language of software architecture. It augments the set of languages we already have.
At that higher level, it does make sense to count the number of boxes (components) on a diagram as a complexity metric, where a constraint is placed on the boxes themselves, e.g. everyone in the room agrees that each box will take less than 2 days to implement.
First, Kudos … it’s an awesome article. It reminds me of entropy in physical engineering.
Our board consists of one software engineer (myself), a person with a master’s degree in communication and another with a master’s degree in psychology with an emphasis in cognitive neuroscience.
We have observed many of the phenomena you observed in your article. Our psychology person observed: “Every time you need to connect to another system, everything slows down and we can’t predict a schedule”.
In your observation, “interfaces are clean, dependencies are few”, I think you have found the key. It’s controlling the number of dependencies and interfaces within each sub system.
I can’t imagine a generalized technical tool to measure the complexity of interfaces and dependencies. I think the solution to identifying the “Cognitive Horizon” may be human, not technical.
We’ve basically gone to a modular approach to manage complexity (similar to the Unix approach you mentioned), defining our software by interfaces and modules. I explain to our board what the software does and how at a high level. Our board member with the degree in psychology is not really listening to what it does, as much as she is listening for how well I understand what I’m explaining. She let’s me know when I’m starting to lose my cognitive grip, before I actually lose that grip.
So, before we actually reach what you call the Cognitive Horizon we determine if it’s time to refactor and re-modularize.
So, rather than changing to new approach, we are focusing on determining ways to identifying how when to use traditional approaches more effectively.
P.S. Our CEO (the guy with the Master’s in Communication) really liked the the term ” cognitive horizon” as a description of what we’ve experienced.
The concept of using psychology to identify the “Cognitive Horizon” is a nice case of lateral thinking. I like it. As an engineer, however, I also like metrics. I wonder if the two can be combined. For example, human behaviour might be a proxy indicator. Given that the “Cognitive Horizon” is both influenced by human and technological factors, i.e. the human ability to grasp complexity and the complexity that is present in a project framed by technology, might we collect performance metrics on people to draw conclusions about the “Cognitive Horizon?” What might those metrics be to rule out other factors? This line of questioning goes directly at the heart of the comment made by Paddy3118 “Where is the science?” There are two ways of arriving at the science: 1) Measure complexity and its impact on projects. 2) Measure “the psychology” as a proxy indicator. Given that neither is likely to support a single unified metric, the proxy approach might be more prone to overfitting, yet potentially more accessible.
Where is the science to back up this “law”?
Thank you indeed for this valid criticism. In the spirit of apologetics, I think I was borrowing on Parkinsons Law which describes itself as a “adage”
which vaguely translates as a “memorable saying which holds some important fact of experience.” As the abstract says, I have observed a dynamic and would like to propose a name. Calling it a “law” is admittedly “reaching” — on purpose. The goal is to stir controversy & draw in comments and other people’s experience. For this to have grounding in hard scientific proof, a whole different standard, one would need access to statistically representative projects metrics of a statistically representative number of organisations. The metrics would have to be of sufficient quality to be conclusive, rule out other factors, etc. Most projects don’t collect this information; most organisations would not make it available if it was collected. And so, like Parkinson’s Law, my experience will at best become an “adage,” but perhaps one shared by others. If it resonates with people, it may inspire the collection of metrics to guide projects, even if those won’t be published…
First of all bravo, great article and well written!
I have been thinking about complexity for some time now on different architectures. There has always been some limit I have recognized as being there and that apparently cannot be passed. This is the first time I have actually read about it in a meaningful way. It really helps me understand it from a different perspective. Thank you!
I try to use a measure that helps me understand whether or not I’m going into a direction where the system is going to become unmanageable. It’s a very crude measure hence I’m not really sure it’s a good one.
When you drop a set of matches, most people can – without counting – easily identify up to 6 or 7 matches, without actually counting them. Once you start dropping more, it becomes difficult and people start to actually count the number matches. There are exceptions to this rule, but that’s what they are, exceptions.
I think the ability to understand a system somehow works in the same way. Understanding a system that only has roughly 7 components is very straightforward. For arguments sake I am using the number seven. I think it’s different for different people. These components can be high level. Each component can then be subdivided into several more components which again can easily understood if their number doesn’t surpass roughly 7. I am guessing interdependencies work the same way. If there are more than 7 “connection points” to other componenets people have to start to think about the connectivity instead of immediately “seeing” it.
Of course, 7 is just a first limit. We should be able to use a system where we might have to think about it first. But that I think depends on what you wish to use it for. If you have to carry the system on your person and you should be able to use it immediately without having to think about it, then around 7 should be the number. Practically speaking, this might be a phone with 7 capabilities on the home screen.
Much like David above, when I run into a system where there are complaints about complexity I try to “lower the number”, by refactoring or remodularizing. This may add a level into the system. But when the first level is too large (let’s say 20 or something), adding a level may actually start simplify it. Of course, if you’re adding more than 20 levels then you haven’t really solved the complexity issue.
Hence I try to use some kind of gut feeling with this number. I know it’s very crude, but it helps me design systems that are – hopefully – still understandable.
Thank you for your feedback and I think you are onto something with the number 7 though that may be yet too large a number. I’ve always found it interesting that acronyms tend to be made of either 3 or 4 letters. 5 or more is rarely seen. This tendency may relate to what is called Working Memory, a form of short term memory that denotes what we can focus on simultaneously. We can recall 6 or 7 digit phone numbers but tend to imprint them in our short term memory by repetition or breaking them down into 3-4 digit groups. So our mind wants smaller groups than 7 even. There is an article here that discusses this http://m.livescience.com/2493-mind-limit-4.html
Another consideration regarding working memory is how it composes across a team. If one team member’s working memory tends to a limit of 4, then what is the team’s collective working memory if there are 5 team members? Is it simply multiplicative and tends to a limit of 20? Is it linear ? Does it tail-off as it approaches a limit imposed by communication overheads? One “accepted” limit on the size of SCRUM teams is 7. See http://scrum.jeffsutherland.com/2003/02/scrum-keep-team-size-under-7.html The similarity in numbers may be a coincidence.
Thank you for everybody’s feedback!
Well written!
I tend to see the Complexity and the Сomplicacy as very different terms, but sometimes we use them “interchangeably”. Each object in Universe is very complex. We just do not go or, more often, not able to go deeper a certain border. Typically even do not need if we able to build our comprehension based on more “affordable” details.
All kind of “natural” things like “the Honeybee”, “the Flower”, everything from, lets say, the wildlife are so cool, because they have millions years of “refactoring” process behind.
We have no such time working even with Agila and Scrum methodologies. Our timeline if more tough 🙂
But the idea for a necessity to measure a level of complexity in some quantitative form is really good one.
To use a metaphor from architecture – Bauhaus not Victorian.
A lot people actually prefer Victorian. Even in places where only Bauhaus can survive. They must be resisted.
Personally I favor refactoring unless you can afford to do the whole damn thing all over again from scratch.
I mention refactoring not because it’s a way to go, but just from a viewpoint that it works “automatically” in “the nature”. That’s why it’s hard to compete with “a perfection and optimality” of all natural things. They are complex, but not “unnecessary overcomplicated”. Its next level of functionality is built on its own “highly optimized” preceding base. Our (humans) intention to “achieve a goal in a shortest time” leaves no room comparable with an optimality and beauty of the natural things.
The below point from the proposed “Law of Tangental Complexity” is the perfectly written summary of the above statement.
“The idiom of a tangent implies that complexity may move beyond our field of view by adding to it in a lateral fashion, i.e. by simply adding breadth of functionality rather than necessarily adding increasingly complicated functionality.”
Interesting thoughts. I’d certainly like to see it put on a scientific basis – and that science won’t be (solely) Software Engineering. SE pretends to grasp this stuff but I think some other disciplines like organisational psychology need to be in the mix.
This is related, too, to something I’ve been musing for a few years: that these issues don’t just apply to projects/systems but to a much bigger ecosystem. One of the reasons that cybersecurity – my present bread and butter – is wildly out of control (and scaring the people who know what they’re talking about) is that the complexity of the internet ecosystem vastly outstrips your cognitive horizon. There are quite a few self-styled cybersecurity experts, but I genuinely think that none exist. Moreover, any hope one might have of getting a project/product back under some sort of control, following you mitigation strategies, largely fails for the domain of ‘cyberspace’. O tempora! O mores!
Adding link to discussions on Google Plus:
https://plus.google.com/+danlentz/posts/YUXTUCZDxU8
Pingback: How Do You Measure System Complexity? | CISQ - Consortium for IT Software Quality