Navigating Numbers: The Intricacies of Goodhart's Law

Dive into the concept of Goodhart's Law as your hosts, Jennifer and Michael, unravel how when a measure becomes a target, it can lose its intended value. Through detailed examples from business, education, and policy, this episode explores the unintended consequences that arise when metrics dictate behavior. Learn strategies to mitigate Goodhart's perverse incentives and recognize the law's implications in the real world.

Creator: rune



Creation Parameters

Prompt: Make a podcast that explains Goodhart’s Law in easy to understand language. Give concrete examples of the law in effect.

Guidance: Limit host banter, go straight to the topic. One host asks questions, the other answers. Use simple language that is east to understand. Give good examples.

Pivate: False



Script

[0:10] Jennifer: Hello and a warm welcome to everyone tuning into Anypod, your go-to podcast for diving deep on a wide array of topics. I’m Jennifer.

[0:19] Michael: And I’m Michael. Today we’re about to tackle Goodhart’s Law and really get into the nuts and bolts of what it means, especially through some detailed examples.

[0:28] Jennifer: Yeah, we’re going to flesh out this relatively simple-sounding law into real-world situations where it plays out. Michael, could you start us off by elaborating on Goodhart’s Law a bit more?

[0:41] Michael: Of course. Goodhart’s Law is pretty fascinating. Imagine you find a wonderful, quiet route to work that saves you time. But if your traffic app starts to advertise it as the fastest route, soon everyone will start using it. As a result, it won't be the fastest route anymore. That's the essence of Goodhart's Law: a measure loses its effectiveness when it becomes an end goal, because people’s behavior changes in response to the measure being targeted.

[1:06] Jennifer: That’s a great analogy to kick us off. So, if we translate that into more formal settings...

[1:12] Michael: Right, let’s take businesses, for example. Imagine a sales team is being measured by the number of calls they make. If their bonus depends on hitting a specific target, say a thousand calls a month, what happens? They might just call without really engaging potential clients, because their aim isn’t to nurture leads; it’s to hit that call count. Quantity trumps quality.

[1:34] Jennifer: Or in other words, the sales metric became the goal, which kind of ruined its original purpose of being a good measure of actual sales performance. That’s classic Goodhart’s Law at work.

[1:46] Michael: Precisely. And it’s named after economist Charles Goodhart. In 1975, he was analyzing monetary policy, and he pointed out how certain monetary aggregates, which were reliable indicators of economic health, stopped being useful once the policy started aiming to control those very aggregates.

[2:04] Jennifer: That’s an interesting bit of history. But let’s bring this to life for our listeners with another example, shall we? Maybe something a little closer to everyone’s lives?

[2:15] Michael: Yeah, standardized testing is a perfect one. Many of us have gone through the education system so we can relate. The purpose of these tests is to gauge student learning, right? But if a school focuses solely on increasing test scores—maybe tying funding or teacher evaluations to those scores—things change. Teachers might start focusing on the material that’s just going to be on the test instead of providing a well-rounded education.

[2:40] Jennifer: Right. And it’s not just about teaching style, is it? Think about the students. They might just cram for tests to get high scores, without really learning the material deeply or learning how to think critically.

[2:52] Michael: Something similar happened here in the UK in the ‘80s with the Poll Tax, or officially the Community Charge policy. The tax aimed at making the costs of local government transparent and linking them to the actual services provided. However, it became the single target for rallying protests against government policies, way beyond local government accountability, leading to heavy opposition and its eventual repeal.

[3:14] Jennifer: And in the healthcare scenario, if we consider patient wait times as another measure, what happens when hospitals or healthcare providers target reducing those numbers at all costs?

[3:24] Michael: You might see things like patients being rushed through their appointments or being discharged prematurely, simply to meet numerical targets. That can compromise the quality of care, which is dangerous – not to mention against the whole ethos of healthcare.

[3:37] Jennifer: These are some solid examples. But I can’t help wondering, is there a way to use metrics without falling into the trap of Goodhart’s Law?

[3:46] Michael: That’s a great question. One thing experts recommend is using multiple metrics instead of just one. In the business scenario, a company could look at both the number of customer interactions and customer feedback scores. It's harder for employees to game the system if they have to score well on multiple metrics, which ideally would mean they’re actually performing better overall.

[4:07] Jennifer: And I suppose evolving metrics could help? As in, not letting any single measure sit long enough to become a target that distorts behavior?

[4:16] Michael: Absolutely! Plus, distancing performance measures from individual reward systems might also reduce the incentive to game the system. The key seems to be having a varied, dynamic approach to setting and evaluating targets.

[4:31] Jennifer: Right. So avoid the pitfalls of Goodhart’s Law by not being overly reductive in our measures and by understanding the complex behaviors behind these metrics.

[4:42] Michael: And that circles back to the essence of Goodhart’s Law. It’s about human adaptability. When we know we’re being measured, we adjust our behavior accordingly – not always in the ways intended.

[4:54] Jennifer: Very true. Well, listeners, I hope we’ve managed to paint a comprehensive picture of Goodhart's Law for you.

[5:02] Michael: Today we've really dived deep into a few examples, and hopefully, you can now spot Goodhart's Law in action all around you.

[5:07] Jennifer: Thank you for joining us on this exploration. It’s been enlightening breaking down these concepts with you, Michael.

[5:14] Michael: And likewise, Jennifer. Listeners, we truly appreciate the time you spend with us here on Anypod.

[5:20] Jennifer: Stay curious and keep questioning the metrics around you. Until next time, this is Jennifer...

[5:26] Michael: And Michael...

[5:28] Jennifer: ...Signing off. Take care everybody!