An organization takes on measurement debt when it implements initiatives without investing in the measurement infrastructure required to validate the benefits delivered by those initiatives. Let's take an example.

Do Chatbots Payoff?

Several organizations have invested in chatbot (virtual assistant) initiatives. They aim to reduce (or stem future growth in) call volumes while preserving or improving customer experience. Those chatbots have gone live and are handling several hundred (or thousand) sessions per hour. But that’s just a vanity metric.

What percentage of chatbot sessions end with a satisfied customer having no further need to make a call that day? An inbuilt survey could help answer this but not all chatbot implementations have it.

Going beyond product analytics

How do we determine the actual impact of chatbots on call volumes? The end-of-session survey would not provide a definitive answer because many of those successful sessions might have just as well been self-serviced through non-chatbot functionality of the website or mobile app. The various digital self-service channels might be cannibalizing each others’ traffic in addition to attracting new traffic. But to what extent?

To really understand the impact on call volumes, it is necessary to step out of the digital silo and head over to customer operations. We might require new, granular reports from the call center to reconcile with what’s going on in the digital self-service channels. It might even require call center agents to add some more call-disposal information. The missing reports are examples of measurement gaps. These gaps need to be bridged in order to effectively demonstrate impact of new initiatives, products, capabilities or features on business metrics. Just reporting based on product analytics available within the digital silo is not enough.

Tech outcomes don't automatically lead to business impact

The chatbot example above highlights the need to understand gaps in measurement infrastructure and the need to invest in closing them if we are to truly validate benefits of initiatives. Else the investments in initiatives are, to some extent, shots in the dark. They might result in delivery outcomes (chatbots delivered, handling N sessions per hour) but not necessarily lead to business outcomes (savings in call volume).

Over-reliance on the business case

Most execs agree on the need to invest in measurement in order to validate benefits, but some disagree. They are often people with decades of experience in a classic, non-digital-native enterprise that used to outsource most of its custom-builds and systems integration. They claim that benefits are assured if the business case analysis is done well. There is no need to further invest in validation of benefits. All we need to do is to execute to plan. Unfortunately, this is not true even though it may be how the classic enterprise has functioned over the years.

ROI all over again?

No. I'm not renewing the case for calculating return on investment (ROI), the ratio of net income to investment. That's a financial metric which is too difficult to determine for investments like chatbots. Besides, it doesn't lend itself to ongoing measurement. Still, we need some way of validating benefits in terms of non-financial metrics at least.  There's a scale of accuracy with which we can gauge if an investment is paying off. Here's one scale (in descending order of accuracy) for the chatbots example:

1. Financial: Call centre cost saved by chatbots

2. Good Proxy: Call volume saved by chatbots

3. More Approximate Proxy: Number of chatbot sessions that customers acknowledge to have saved them a call.

4. Vanity Metric: Absolute number of chatbot sessions

Settle for #2 or #3 depending on your situation with the understanding that #3 is susceptible to the cannibalizing effect described earlier. Although #2 takes a bit more rigor, it is easier than #1 and we can approximately estimate #1 with #2.

Measurement Debt

An organization takes on measurement debt when it implements initiatives without investing in the measurement infrastructure required to validate benefits delivered by them. This debt is incurred not as a choice or a strategy but as a consequence of neglecting measurement.

What's the principal that's borrowed? We borrow the team capacity/bandwidth/budget that should have been used to evolve measurement infrastructure alongside functionality. We use the borrowed bandwidth to add more functionality unsupported by measurement.

What's the interest payable on measurement debt? Interest payments consist of:

1. The portion of investments that could not be saved because we did not have adequate measurements to tell us that they weren’t delivering the benefits expected.

2. The lost business upside from not doubling down on successful investments because again, we did not have adequate measurements to tell us that they were delivering more benefit than expected.

3. The cost of all future sub-optimal decisions made without the benefit of learnings that could have resulted from the having the measurements in place.

Over the years, organizations have learnt the costs of unaddressed technical debt the hard way. They now set aside some budget and team capacity to repay tech debt on an ongoing basis. Strong engineering departments make sure of this. They also make sure to teach good technical practices that help slow down the build up of tech debt.

Should we wait to learn the hard way again before acting on measurement debt? That might have severe business consequences.

Critical thinking Exec Sponsors needed

Who should take ownership of addressing measurement debt? And how do we fund it?

Ownership should rest with those who are (or will be) most affected by measurement debt. This debt affects business and product leaders more than technology leaders but it affects CFOs and COOs the most. This is because the investor community looks to COOs and CFOs for answers to how investments have paid off. Their questions may not be so pointed during phases of rapid growth but they are inevitable as growth plateaus. At that point, it would be too late to begin addressing measurement debt because it will take several quarters to make any difference. Besides, getting into the habit of validating benefits is a cultural and behavioral change. It is a muscle that gets stronger with exercise. The time to begin is now.

My specific recommendation comes in the form of a broader solution called Business Retrospectives. It is a comprehensive, new method to help clients validate benefits of investments and thereby progress from tech outcomes to business impact. As part of its rollout, it requires a COO, CFO, or Chief Performance Officer to sponsor a Measurement Improvement Program (MIP) to address measurement gaps like the ones described above. Gaps identified for approved initiatives enter the MIP backlog. There’s a lot more to Business Retrospectives. Here’s an outline.

It might seem ironic to make a new investment to help validate the benefits of investments that seem necessary on paper. But this is really about investing in a data-driven culture of learning and continuous improvement. This type of learning is at least as important as the one offered by the "Learning & Development" group in a classic enterprise.

How do you make it happen if you are not a decision maker? Pitch it to critical thinking executives in your organization—those that don't wait for big brand advisory folks to recommend it after it has become blindingly obvious. To begin with, show them the costs and benefits of addressing measurement debt for a couple of important upcoming initiatives. Challenge them, if needed, to explain why business metrics haven't moved as much as projected in the investment business cases of recently delivered initiatives. Ask them if they would be comfortable with a similar challenge from investors or the board.

Conclusion

Measurement debt stands in the way of validating benefits of investments in initiatives. Without validation, we won't know why business metrics don't improve in proportion to investments. Sooner or later, this will put the execs answerable to investors in a tight spot. They must act now and sponsor efforts to address measurement debt. That's the way to show seriousness about a work culture based on learning and continuous improvement.