That's not a mistake!

2024-02-26

You do something, and the result is bad, really bad. Did you make a mistake? Not necessarily.

Decision vs. outcome

Imagine there's a random number generator with outputs between 1 and 5, and you have to bet on whether the next number will be odd or even. What would you do?

Obviously, you would bet on odd numbers because there are 3 possible odd numbers and only 2 possible even numbers.

After you place your bet, the next number is generated, and it's an even number!

Did you make a mistake by betting on odd numbers instead of even numbers? Of course not. In other words, if you have to bet again, it's better to bet on odd numbers again.

Outcome bias

Our naive tendency to judge individual decisions based on their outcome is called outcome bias, and we'd do well to be aware of it and try to control it.

To practice, you can play a board game that mixes skill and chance, such as backgammon, while being aware of the outcome bias. It's also fun :)

Lessons learned

Why did I think I should talk about it?

It's because I've noticed that some people think that lessons learned in projects are all about mistakes. This is not correct because of two reasons:

• Some lessons are positive; for example, "We thought that the long reports the customer wanted us to send were inefficient. So we still sent them the long, detailed reports, because otherwise they would have said we weren't doing a good job of reporting, but we also added an extra one-page report with all the highlights. Gradually, we learned that everyone actually reads and understands the short report, which helped our communication a lot. It's probably a good idea to do the same in all projects."
• Negative results are not always about mistakes. In fact, they usually aren't. For example, "Since it was the first time we had a project in Artopolis, we asked around and found that their primary method of communication was instant messaging rather than email. So we chose that as our main communication channel to fit the environment, since most of the people involved in the project were local. However, we soon learned that instant messaging is a terrible solution for communication, because different topics are not separated from each other, it's easy to miss topics, it's difficult to search for old topics, it's difficult to archive them, and so on. It's just a bad idea to use instant messaging as the primary method of communication in projects. I have no idea why they do it that way in Artopolis!"

When done properly, less than 10% of the lessons would be about mistakes.

The reason

The reason we shouldn't use the outcome of a single event to judge the decision is that there are many uncertainties and unknowns, and we have to make our decision based on the limited information we have.

Now, there's a complication: Some uncertainty is inevitable (think of the uncertainty principle in physics), but that's not all, and it's mostly about our lack of information, which we can change by spending more time and money.

For each decision, we can continue to spend more and more effort gathering additional information and then make the decision with a near-minimum level of uncertainty. But this has to be justified: If you have to choose between two brands of paint for the fences, you're not going to stop working for a few weeks, spend hundreds of thousands of dollars, and run all kinds of tests in a lab to see which paint is better, because the expected value of the negative outcome doesn't justify it. It's better to just pick a paint with the information you have and move on. If it turns out to be a bad paint, that's fine, we can repaint it with the other paint by spending a little time and money.

The framework

So, when we get an unexpected/important/sensitive outcome (either negative or positive), we need to ask ourselves two questions, one to address the correct concern and one to address its meta-concern:

• Did I make the right decision, given the information available?
• Did I spend a justifiable amount of effort gathering information to make that decision?

If you do this correctly, you may find actual mistakes that led to positive outcomes, cases where you realize you spent too much effort on a decision that wasn't worth it, etc.

Why does it matter?

We shouldn't do anything without a clear purpose (NUP5). The purpose of lessons learned in projects is not to find mistakes but to make better decisions in the future. What's discussed in this article is one of the things that can help you make better decisions and, therefore, would be reflected in your implicit or explicit lessons learned system.

— the end —