Like Al Pacino in The Godfather, I can’t seem to escape the world of governance indicators no matter how many job and life changes I make (“They keep pulling me back in!”). Given the decades I spent navigating the murky universe of squishy concepts like governance and corruption, last week’s bombshell news that the World Bank was abandoning its famous Doing Business report/dataset came as a reminder that yes, it’s still a bit of a mess out there in Indicator Land.
For those less tuned in to this admittedly niche topic, here’s the Reader’s Digest version of what happened. After a few years of outsiders calling into question the accuracy and veracity of Doing Business - ostensibly the world’s most influential dataset measuring a country’s pro-business regulatory environment - the Bank hired venerable white shoe law firm Wilmer Hale to get to the bottom of whether Bank staff were manipulating Doing Business data in some fashion.
While nerds should read the full Wilmer Hale report, the short answer is, “Yes, they did” and in particular sought to change data to satisfy concerns from China and other influential World Bank donors that their rankings in Doing Business weren’t high enough. It was ugly political pressure from the very top, from the then-World Bank President’s office as well as now-IMF chief Kristalina Georgieva, who comes off looking particularly bad in the report (going so far as to visit a Bank staff’s house on the weekend to ensure the final Doing Business report for that year contained the doctored data she had ordered staff to include). Scandalous stuff, and the Bank’s board moved swiftly last week to terminate Doing Business entirely.
Besides the obvious and banal hot takes (“the World Bank is influenced by countries that give it money!”), I find a few dimensions of the imbroglio particularly interesting:
Countries really do care about their rankings (or, at least, some rankings)! One silver lining of this debacle is proof positive that indicators can and do “work” to influence countries and governments. What I found most fascinating about the Wilmer Hale report was the degree of concern and anxiety expressed by Chinese officials to World Bank brass about their placing in Doing Business. While that concern was never going to automatically lead to the sorts of structural and regulatory reforms sought by the Bank (and others) in China, it provides another powerful anecdote in favor of the (perceived) power of quantitative measures and ratings. We know from the work of the now-defunct Governance Data Alliance that certain data publishers - specifically UN agencies like the World Bank and OECD along with leading consultancies such as McKinsey - carry real weight in shifting opinions and sometimes even policy inside of public sector institutions. Put another way: indicator exercises are not a waste of time and can generate real leverage and political space for reform.
Pseudo-empirical measures of complex social phenomena remain methodologically fraught. Another anecdote that stood out for me in the report was the relative ease by which Doing Business staff were able to adjust countries’ relative rankings through small amounts of data manipulation. At the core of this ease is the nature of many of these “expert assessment” exercises. Despite the voluminous amount of coding handbooks and methodological guidance provided to Doing Business’ country researchers, the final decision of whether an indicator for Country X is scored a 3 versus a 4, for example, remains at the end of the day a bit of a judgment call. Bank staff found that they could make those small changes within the constraints of their robust methodological framework while simultaneously maintaining a defensible rationale for why 4 was suddenly a more accurate score than 3. It’s a sobering reminder that “measuring” things in this world is always a risky proposition.
The ripple effects of Doing Business going away will be significant for other data producers. I can’t even begin to wrap my head around how many ratings agencies, ESG data providers, and other purveyors of aggregated metadata had their minds blown by this news. There is a dangerously large industry of data providers who do little more than vacuum in lots of squishy data like Doing Business and then crank out (and often sell) even squishier metadata, cranking suspect source data through methodologically questionable data aggregation meat grinders. If you’re relying on data like that for actual decision making or investment choices in the real world, this scandal should be a wake-up call that you’re playing with fire. Glossy numbers /= truth.
There’s an opportunity for civil society data producers in all of this. How much trust will outside observers place in data produced by agencies like the World Bank moving forward? I can't imagine it’s too much. If I were still leading a civil society organization that cared about quality data in these spaces, I’d be getting on the phone with every UN agency I could find offering to partner with them to provide independent third-party verification of the numbers they produce moving forward. Before this scandal that would have been a laughable proposition to most UN agencies; now, it might be essential for the survival of their data projects. Perhaps future civil society efforts in this area might be best oriented towards strategic partnerships with The Big & Influential Data Producers (World Bank, IMF, OCED, McKinsey, etc.) rather than cranking out their own competing datasets.
I’ll continue to munch on my popcorn over here. But wow, what a week it’s been in Indicator Land.