The next piece of software is to be developed by 2 teams and they now have to answer how long it’s going to take. Team A says 5-8 months and Team B 6-9 months. But the department head is under pressure to deliver a date, so he takes 7 months as the average of the 2 teams. How can the Monte Carlo simulation help?
Daiany Palacios is a DevOps Analyst/Engineer at Kuehne + Nagel. She gave a talk at Codemotion Rome 2019 about why average-based plans fail on average, then explored the psychological problems we humans have dealing with probabilities as a possible reason why we keep falling into the averages trap.
A world of averages
We live in a world where we have headlines of averages in the news, in the stock market in the average debt of the country. And unfortunately, software development plans are not the exception.
According to Daiany,
“The problem is, whenever an average value is used to represent an uncertain quantity that ends up disturbing the results because it ignores the variations. There are simply too many uncertainties too much variation so that it can possibly work like that. Instead of taking a single value, we change our mind and act on the distribution better than on a single value. So the idea is to create a distribution of outcomes. possible values and the likelihood of them happening and evaluate that instead, in order to get a more accurate forecast.
The Monte Carlo simulation
The Monte Carlo simulation is a computerized mathematical technique that allows people to account for risk in quantitative analysis and decision making. The technique is used by professionals in such widely disparate fields as finance, project management, energy, manufacturing, engineering, research and development, insurance, oil & gas, transportation, and the environment.
Monte Carlo simulation furnishes the decision-maker with a range of possible outcomes and the probabilities they will occur for any choice of action. It shows the extreme possibilities—the outcomes of going for broke and for the most conservative decision—along with all possible consequences for middle-of-the-road decisions.
According to Daiana: “It’s a mathematical tool that allows us to make data-driven decisions in a better way. It takes historical data as an input, then runs a large number of random simulations on how items under similar circumstances are going to be in the future. And in the end, we have all those random simulations have a kind of distribution, that is what we evaluate it. We’ve been using that for the past three years in our teams, my department. And we’ve had really good experiences with it to such a such a point where we just don’t do item based estimations anymore.”
In terms of practical application, Daiana shares how it works:
“Whenever a product owner or product manager comes over and says, ‘This is the new feature you need to implement’, the first thing you do is to define the number of working items. Then there’s task breakdown where you write down the steps that needs to be done in order to implement the requirements. We count the items instead of estimating.
How does the Monte Carlo Simulation work?
Daiana explains that their team use JIRA, where the workflow system records how long it took to finish one item or task, from the moment you started until when it was finished. Then they “export how long it took in a chosen prior timeframe from the past, say, six months, do a data export, and then we give it as an input to the Monte Carlo simulation. Then we run it and then evaluate the results.”
The Monte Carlo simulation is akin to rolling a dice: because it’s uncertain, so you cannot possibly know how long some items in the future are going to take. So the Monte Carlo simulation basically rolls the dice for you, thousands of times, using the data provides as an example.
So a random item from that set of historical data that you provided, and pretends that it’s how much an item from the future is going to take. And again, and again, thousands of times 10s of thousands of times until a plausible statistical prognosis is created.
And “since the base is the true historical data from our team, from our performance, and it’s pretty reliable the result that comes out afterwards. Thus, it’s a probabilistic forecast. These kind of forecasts are rather hard to accept because you go to your project manager and say, Okay, we’re going to be ready with an 85% probability by 25th of February or earlier. Because it means acknowledging that it means a change in the mindset. It means that we have to think probabilistically another deterministically. That means that we have to acknowledge the possibility that there is more than one possible outcome in the future.”
How does the Monte Carlo simulation deal with uncertainty?
The matter of uncertainty is handled beyond the initial forecasting by the Monte Carlo simulation. The simulation is repeated “as soon as we have more information because, at the beginning, you don’t know a lot, but then the more you progress, the more you know. So the uncertainty reduces.
Thus, Daiana and her team do continuous forecasting. “Continuous forecasting means that we get used to the idea that that date might change in the future. It’s a living forecast that we update, as we have more information, it will change probably not so extreme. But it helps also for us psychologically, to get used to this idea that we have a probability there because we are repeating it constantly.
We sit together with the product owner regularly and run the simulation.”
The impact of the Monte Carlo simulation
According to Daiana, the impact of the simulation is significant. She works in a very traditional workplace where predictability is far more important than the pure speed of delivery.
“Since we started doing continuous forecasting, we noticed that the more disciplined we are with our development process, the more reliable the results of the Monte Carlo simulation are. For example, we focus more on finishing tasks before starting new ones.
