Why Comparing Performance with the Past Misleads in Complex Systems

We often judge things by comparing them with past results. But when assessing effectiveness in complex systems, benchmarking against the past is not always valid and frequently leads to erroneous conclusions.

This matters most for top managers and business owners when they evaluate how well particular tools, or particular people, have worked.

Take company X. Last year its profit grew by 20%, and its revenue by 25%. On these figures alone, everything looks splendid, and the company appears to be performing well.

Over the same period, prices for the goods the company sells rose by 15%, while the cost of intermediate goods and wages rose by 10%. Last year the company also acquired a competitor whose revenue was equal to 25% of the company's own. Once we see this additional context, it is clear that things are in fact bad. The company lost part of its customers in the acquisition (around 15%), and its cost of goods rose faster than the market's.

Or consider another example. A company rolls out new software in some of its stores, but at the end of the quarter sales have fallen by 10%. It would be easy to draw the wrong conclusion about the software's effectiveness. But we have not asked how far sales would have fallen without it. To assess the new software, we need to look at sales dynamics in the outlets that do not have it. If the decline there is greater than 10%, the new tool is effective overall.

These are highly simplified examples, and in complex systems it is almost always essential to analyze the influence of individual factors.

Assessing effectiveness is especially hard in a growing market with high profits. Yet when a crisis hits, or new technology-driven companies enter the market, adapting quickly to the new reality becomes extremely difficult.

The same applies to the macroeconomy and to any industry: to understand how effective a given person or tool is, one cannot simply compare aggregate data. Deep analysis is required.

Why comparison with past results often leads to wrong conclusions

In business management, a simple method of assessing effectiveness is widely used: comparing current results with those of previous periods. It seems logical, but in complex systems this approach gives a distorted picture and leads to poor management decisions.

Owners and top managers should keep this in mind above all when analyzing the introduction of new tools, business processes or management decisions.

Example.

Company X shows profit up 20% and revenue up 25%. At first glance, these are good results.

But:

Prices for the products it sells rose by 15% over the period.

Cost of goods and the payroll rose by 10%.

During the year it acquired a competitor whose sales were equal to about 25% of the company's current revenue.

At the same time, the company lost about 15% of its customer base.

And its cost of goods rose by more than the market average.

Taking all of these inputs into account, it becomes clear that the company's operating efficiency has not improved but, if anything, deteriorated. The growth in profit is largely the result of external factors and changes in structure, not of internal efficiency.

Another case.

New software is introduced in some stores. A quarter later, sales are down 10%. Formally, a failure.

But without analyzing a control group, where the software was not introduced, no conclusions can be drawn. If sales in the other outlets fell by 15%, this means the new tool had a positive effect, and the decline would have been even deeper without it.

Conclusion.

Assessing effectiveness is not just a matter of comparing numbers. In complex systems it is important to consider the context, external and internal factors, and the existence of alternative scenarios. Without this, management decisions become reactive and often mistaken.

Macroeconomics, industry indicators, internal transformation: all of them require in-depth factor analysis. A simple "before and after" comparison does not work in these conditions.