About waterfalls, agility, research and paradoxes

Some time ago I’ve watched a funny video on personalities that actually has absolutely nothing to do with this article, except one funny notion that I wanted to use here:

this person was a Greek philosopher,
fantastic job actually,
you sit on a stone the whole day long and you think of stuff

That person, or actually two of them, was Meno and Socrates. No idea if they were sitting on stones thinking actually, but they managed to get some pretty cutting edge research done. They were namely arguing about Virtues and came up with a paradox statement:

And how will you inquire into a thing when you are wholly ignorant of what it is?
Even if you happen to bump right into it, how will you know it is the thing you didn’t know?

Possibly they didn’t use English to formulate it, but still, the quote should be close enough to capture the idea (badum, tss)

But what is so remarkable about it?

Funnily enough, this paradox draws on the axis of uncertainty, or being informed about something. The closer one is to one edge of it (the less one knows about a topic), the more difficult it becomes to actually even ask the right questions. Not even mentioning to answer these.

Let’s have a walk down the axis

How it feels being on the leftmost of this axis, one can imagine when talking to a scientist doing cutting edge research:

William Gladstone: But after all, what is the use of it?
Michael Faraday: Why, sir, there is every probability that you will soon be able to tax it!

Mind it, Faraday supposedly said it upon demonstrating an experiment with electricity. He just didn’t know what the possible applications of it could be. I guess it wouldn’t be also such a wild presumption to expect that he had even less clue how much time it would take to invent the experiment when he got interested in the topic itself. When will a scientist be done with research? Once he’s done.

How it feels being on the rightmost of the axis? Try imagining buying a piece of furniture from IKEA. Do you expect to have all necessary parts inside the box? Do you expect to have a manual how to put things together? Do you expect the assembled thing to ‘work’? Do you expect it to be warranted that, should it not work, that you can just give it back and get a replacement?
So, pretty much ‘everything’ is known. Including how much it will cost, how much time it will take to assemble, deliver, etc.

What wild-lands lie in-between of those two?
Places where you have some information, but not exactly all that you need. So some things run predictable, others require experimenting. The closer to the left side, the uncertainty – the more experimenting.
What is known here? Hah! Very good question! But at least one can approximate.

How does it look like in the IT world?

This very situation occurs day-in day-out in IT.

Ask a designer when he will come up with a new, revolutionary, nice, user-friendly design of a user-interface …. you could just as well be asking Faraday about the applications of his freshly discovered electricity. Unclear specifications mean unclear time schedule.
The peculiarity is when people expect such creative tasks to be predictable and plannable and try to squeeze those into sprints in Scrum instead of just monitoring those in a Kanban approach.

I would venture a guess that Project Managers’ dream come true are the lands of the Waterfall – where (almost) everything can be perfectly planned ahead. How things are built, how they are tested, how much time they take, how much money they cost, etc.
However for it to be like this, the uncertainty about the information, requirements, etc. must be minimal. The remaining uncertainty can usually be handled by hand-ruled adding of ‘safety buffer’.

In between, where nobody, the customer, the developer, tester, product owner, really NOBODY really knows what the end product will be… one can only ‘endlessly’ improve.
Make a prototype. Verify it. Think of improvements. Make another prototype. Verify it. Think of more improvements ….
Good old, and so buzz-worded, Agile.

Solving the paradox

The interesting bit is – the Agile approach is actually ‘the solution’ to the Meno’s paradox and happens underlies the paradigm of research and learning:

  • Make a hypothesis
  • Construct a model / prototype
  • Verify the hypothesis – make an experiment
  • Find areas of improvement
  • Iterate the points using improved hypotheses until desired accuracy is achieved

The clue here is – without formulating the hypotheses and measuring the results the improvement circle is broken and does not deliver as much value.

How coding and testing is tied to the Heisenberg’s uncertainty principle

In this article I would like to show one interesting connection between IT and physics that I’ve stumbled upon, which is to me quite fascinating. It will have some formulas and relate to terms from physics, but I hope you will not shy away because of that.

For all those who don’t exactly know what the fuss about Heisenberg’s uncertainty principle is –  read about it here, or better – enjoy this fantastic video by 3blue1brown.

In short, the principle says that:

The more precisely one property of a system is known,
the less can be known about its conjugate property.

If you’re now scratching your head wondering about what conjugate properties are – again, read here all about it (if you like maths). This mentioned conjugation relation causes that the order of executing the operations becomes important. Okay, I admit, the last sentences sounded dry… operations, ordering, conjugation, uncertainty… TL, DR, See you… but if you haven’t clicked away, then bear with me, please. Things should become strikingly clear now.

Fancy recipe for a disaster …

Let’s imagine two simple operations:

  1. X = putting on ski
  2. P = riding down the slope

Even a kindergarten child can imagine the consequences of doing #1 first, #2 later and compare them to the results of doing #2 first and #1 afterwards instead…

  • ‘put on skis’ and then ‘ride down the slope’ = fun experience
    X * P = Fun,
  • ‘ride down the slope’ and then ‘put on skis’ = interesting thing to imagine, but not necessarily a fun experience
    P * X = Disaster,

As insane as it may seem now, it does bring value to measure the difference between these two:
X*P – P*X = Fun – Disaster, or in short notation [X, P] = Fun – Disaster. By the way – you have just encountered and understood a commutator – here’s a more scientific description.
In human words, the order of executing of these operations makes all the difference between fun and disaster.
Well … that’s not exactly rocket science. Everybody knows that. As useless as it may now seem – we’ve got ourselves either a fancy recipe for a disaster, or:

… a recipe for how to avoid a disaster

It’s just as obvious when one substitutes the operation names in the following manner:
P = altering the code and X = testing what the code does.
After all, if we first code, and then test, we will get the actual information about what the code actually does. If we instead first test, and then code – weeelll, then we only know what our code did before we changed it. That last thing is rather useless.
X*P – P*X = difference between knowing what the code does in the newest and the previous code version.

Still not rocket science, right?
In physicists vocabulary, this is called the non-commutation of variables – read here if you want some ‘technical’ details.
In human digestible terms – if operations commute, there’s no difference in results with regards to order of execution, if they do not –  the order of executing them is important.

But what about the uncertainty?

The Heisenberg’s principle

mentions uncertainty. But wait. Uncertainty about what?
Those of you who are familiar with it, already know that it just says that there is a limited amount of knowledge one can have about a system under investigation. The more one knows about one conjugate property of the system, the less can be known about the other. All that because of the limitation: how one can interact with the system. As one joke nicely frames it:

Policeman: Are you aware that you were speeding at 150km/h?
Prof. Heisenberg: Well great. Now I have no clue where I am.

How does that translate into the IT world?

Let’s consider the following fact: both coding and performing tests on that specific code cannot happen simultaneously. One must be executed after the other. Both however take place in the same project during the same time frame.

The more time is taken for coding, the less time remains for executing tests. = The more we know where the project went, the less we may know where it actually is.

It is one of the typical failures during sprints – to have insufficient time left for testing of the implemented features. How can testing (position measurements) influence coding plans (momentum measurements)? Imagine this type of situation, and I guess all will be very clear: A day before the sprint ends a tester informs the Product Owner that a blocker bug has been found in this sprint’s batch of features… worst case chaos ensues, next sprint scope is affected and … and … and …

But for now let’s just note that:

The outcome of a test (position measurement) may influence the code development plans (momentum measurements).

As simple as it may sound now – there must be balance between coding and measuring what the code really does. Otherwise one risks misleading the project, or if you like the previous analogy – riding down the slope before one put the skis on.

The more observant ones, who are now asking themselves – what about this fancy right side of the inequality – I hope to explain it in upcoming articles 😉