Today, we conclude the interview with Dan Heath. In this part 2 of our conversation, we dive into respect and heroics for efforts of individuals who have prevented a team member from having to save the day or come to the rescue when a system fails.
In today's episode, we conclude the conversation with Dan Heath, author of a new book , Upstream.
In this part 2 of the interview, Dan talks about quality of leadership and giving more praise to those leaders who solve for disaster before it happens and reduce the need of heroes to save the day when the system breaks.
If you missed part 1 of the interview with Dan, be sure to go back and check that out. in which we kicked off the with Dan, discussing preventative work over reactive work among teams.
Whether you’re working on a personal project or managing your enterprise’s infrastructure, Linode has the pricing, support, and scale you need to take your project to the next level. Get started on Linode today with a special $20 credit for listeners of Developer Tea.
Visit: linode.com/developertea and use promo code developertea2020
P.s. They're also hiring! Visit https://www.linode.com/careers to see what careers are available to you.
If any of the people listening to this are intrigued by this idea of kind of fumbling forward toward a learning organization, I'd recommend you check out the work of a guy named Steve Spear, who has devoted his work in the past several years to studying cultures like Toyota or Navy submarines or places where they absolutely have to get things right and have to get things better on an ongoing basis. And all of what that means from a cultural transformation point of view. The good news is there are a lot of people who have blazed this trail. There's a lot of really excellent organizations and institutions that have taken one step after another on this road to quality improvement. And they're in a place now where we can learn from them and start adopting some of these practices. As we learned in the last episode, it's not always easy or intuitive. To think upstream of a problem, to be preventative by nature, to design organizations and incentives around trying to solve problems before they ever happen. But it can be incredibly rewarding to our efforts. In today's episode, we continue this discussion with Dan Heath. Dan is the author of so many incredible books, including being the co-author of two New York Times bestsellers, Made to Stick and Switch. Dan is also co-author of the book Decisive. This changed the way that I think about decision making. But in today's episode, we're discussing a new book that Dan has written called Upstream. I encourage you to go and check out the book, of course. But also listen to what Dan has to say in this episode about what it means to solve problems before they happen. If you missed the first part of the episode, I encourage you to go back and listen to that. Because there's so many good. And pieces of wisdom that Dan has to offer to us as engineers. Thank you so much for listening to today's episode. Let's jump straight into the second part of my interview with Dan Heath. In the book, you get into these details a little bit more kind of mechanically, specifically talking about, you know, uniting people. And what are the changes actually that you need to make to a system? How do you determine some of those things? Finding leverage. And one thing I'd like to talk about specifically is how do you know when this is succeeding? I mean, in the point of, you know, when we're talking about the the children that are that are drowning, it might make sense that if you had a rate of children drowning one every five minutes and that drops to one every 24 hours, then, you know, that might make a good measurement. But it's not always that easy, right? No, it's not. And I think the reality is we live in a world where. In the fictitious parable world. I mean, my guess is that in normal corporate America, what people would be measured on is is the speed of rescue. You know, and in fact, there's there's an example in the book that I think illustrates this well. It's about Expedia, the online travel site. And back in 2012, this guy named Ryan O'Neill is studying some data about the call center at Expedia. So if you book a flight or a hotel or something and something goes wrong with your reservation, you can call the 1-800 number. Right. What he found made his jaw drop. He found that for every hundred customers who booked a transaction, 58 of them ended up calling the call center for help, which which would pretty much seem to nullify the whole point of having an online self-service travel site. And so he starts digging in to figure out why are so many people calling us? And the number one reason people are calling, I mean, to the tune of 20 million calls in 2012 was to get a copy of their itinerary. That was it. 20 million calls. Can I get a copy of my itinerary? And so he and his boss just they're like, this is madness. We've got to do something about this. And they make the case to the CEO to create a special team to work on this. And they do. And the technical solutions, as you might expect, are pretty simple. They change the way they send. It's not like they forgot to send the itineraries. They were always sending them. It's just they would end up in spam or customers would delete them thinking they were ads or that sort of thing. So. They changed their strategy and emailing. They added self-service tools on the IVR and online and so forth. And they basically took 20 million calls and whittled them down to zero. So from a from a technical perspective, this is a trivial problem. But I think what's interesting about this story is, is why this problem got to this level. Like you would think that there would be an alarm bell that would go off somewhere once you reached like your your three millionth call for an itinerary. And like people would start to take this seriously. But but the deal is that Expedia, like like virtually every other company, has to organize itself or chooses to organize itself in in silos. And so you've got a marketing team whose job it is to to attract customers to Expedia versus Kayak or someone else. And then you've got a product team whose job it is to make the site so smooth and intuitive that the customers are funneled toward a transaction. And then you've got. The tech team that makes the plumbing run and keeps uptime high. And you've got the call center that's trying to minimize, you know, the response time to field a customer issue and to keep them happy via net promoter score or something like that. And from a silo perspective, like all of those goals make sense. But but the problem is when you ask a very basic question like whose job is it to keep customers from needing to call us? The answer was nobody. Yeah. Yeah. And it was even worse than that. Like none. None of those silos even stood to benefit if the number of calls went down. And so that's something I think that's really interesting about upstream problems is that it's often very easy to find owners for downstream problems. Like your house catches on fire. It's the fire department's problem at that point. Like it may not be an easy problem, but it's an easy problem to define an owner for versus if you flip things around and you say, whose job is it to keep customers from calling? Or whose job is it to? Keep your house from catching on fire. Well, that's a very different thing. And it might require the effort of lots of people across different silos that often these challenging upstream issues fall in the gaps between silos. So if you're facing an ongoing problem in your work that you think is preventable, my guess is it might require some new kind of solution. integration that, that the organization of your company makes counterintuitive or makes a bit more effortful than it, than it probably should be. Hmm. Integration and incentive are the two words that keep on coming to my mind here. Integration is, you know, this idea that you've got these two jobs. And if you were to combine them in some way or find, you know, you can imagine that it might be somebody's job to make. Uh, material that is flame resistant right and that's that seems like a reasonable downstream effort we're going to make material that that doesn't catch on fire very easily and then it's easy to see that firefighters are going to fight fires but is there a connection between the material making and the fighting the firefighting where is that kind of if you were to travel up the tree you know where is the the shared branch right where uh that that cares about both of those things yeah this is this is this is a really important question and it's something that i talk about in the book under the framing of of accountability that that one of the barriers to us thinking upstream is is a lack of ownership uh so to your point i mean at expedia the reason they got to 20 million calls in in a year is because no one owned this that it wasn't obvious who should step up and and be the owner and in fact it took a ceo level intervention to crack it but it's literal owner in some ways right yeah i mean you just keep having to work your way up the chain until you get to someone who's not stuck in a in a silo that defies collaboration but it's it's not always that bad i mean that's a pretty extreme story to have to go all the way up to the top of the chain to get a resolution uh and like one of the examples i give in the book that this is kind of a long story i'll try to tell a a really reduced version of it but there was a single pediatrician in in tennessee guy named dr bob sanders who is probably responsible more than any other human being for there being mandatory car seat laws i i've got a couple of little girls and it's absolutely unthinkable to me that i would go on some long road trip without them being in car seats that's just my world you know but but it wasn't that long ago you know i.e in the 70s when it would have been perfectly normal for a couple of toddlers to be flopping around in the back seat not only not in car seats but maybe not even in seat belts um and as a result of that there were just huge numbers of kids being maimed or killed in car crashes and so the chain of of events here was a couple of of advocates wrote a notori not notorious it's a wrong spin it was like a a famous article in the journal pediatrics that called on pediatricians to own this issue that they were not going to be able to do it in the issue of car safety because they were saying if you really are in the business of protecting your uh patient's health your patients being the children that you serve they presented the damning statistics that included this one that more children in that era were dying inside cars than outside them from all other sources put together so they said if you really are are concerned with the health of your patients no you're not in the car with them when they're auto manufacturer no you're not an auto safety advocate but you are probably the number one person in their life that's concerned with their health and you need to pick this up and dr bob sanders that kind of hit him in the heart he was like by god you're right i do need to do something about this and and i'll spare you the whole chain of events but he got interested in politics he starts crusading within the state he uh eventually manages to get passed in the tennessee state legislature the first state act to mandate the use of car seats in cars and then within a matter of years the other 49 states followed but if you notice something about that story that's kind of weird this is a big deal this is a big problem they're high stakes and yet the leadership of this bob sanders and his colleagues were basically kind of volunteering right this is something that's so strange about the distinction between downstream and upstream that downstream happens action is almost obligated you know the kid drowning in the river it obligates our work you know the website goes down we're obligated to respond but but things on the upstream side of the equation even though they may be of equal or larger significance a lot of times it requires somebody to kind of raise their hands and say hey we didn't create this problem but we're going to be the ones to fix it um and and so i think that's a really important thing to think about and i think that's a really important thing to think about and i think that's a really important thing to think about so maybe some of the people listening to this that there's some problem that that you're facing in your company or maybe even in your community um that lacks an owner maybe you're the person to be that owner to be that dr bob sanders character it seems like in emergencies we have emergent ownership right that's kind of the the whole idea is it imposes itself on us and it doesn't even necessarily have to be you know a dire situation uh to be able to do that so i think that's a really important thing to think about and i think that's a really important thing to think about uh to be able to do that so i think that's a really important thing to think about considered an emergency in a given scenario an emergency can be something quite small but it's like you're saying it it creates the sense of obligation to anyone who's around or anyone who has that particular skill set to solve that problem and in some ways that seems directly tied to that heroics conversation we were having earlier uh the idea that hey you know i'm in i'm in this spot and i suddenly became an owner and you hear stories like my instincts kick in right and it seems something again very core whereas like you're saying there's there's some level of volunteerism to solve upstream yeah now here's here's the question that i it may be a difficult question to ask here how do organizations create a more systematic approach you know i i hear very often uh in in many of the companies that i've been involved with and people that i've talked to who work as developers the kind of organizational leaders are asking people to take ownership it's a this constant struggle between people who don't know you know what level of ownership they're quote allowed to take right and then leaders who wish that their people would take more ownership how do we design organizations where ownership becomes uh maybe more accessible or maybe the incentives are better aligned so that the average person is more likely to volunteer i think a lot of these issues especially to do with organizational culture are ultimately emotional that that people are very sensitive whether we know it or not to what what earns you respect and what earns you acclaim and so so part of the answer has to be we have to do a better job of uh heroizing the people who prevent the need for heroics i actually heard from a reader uh last week who was sharing an anecdote from somewhere they had worked where earlier we were talking about this cycle of heroics and they were saying you know that they they dealt with a lot of that to the point where it seemed to be feeding on itself and so they made a conscious effort to change that and they created something they called the smoky the bear award for people who prevented crises and i thought that was just just a great way to kind of get people to do that and i think that's genius you know this idea that we have to be careful what we reward um not in terms of literally financially although that would be nice too if there's some way to manage that but but even emotionally like who who is praised who is held up and and next time we're about to praise the hero who saved the day could could we at least pay some attention to hey you know just to kind of tune into current events i would love to know which organizations were actually pretty well prepared for this pandemic and and and obviously it's going to differ by sector the tech companies are going to be better prepared than others but but if you could control for sector wouldn't it be interesting to see like was staples or office depot better prepared or was lyft or uber better prepared because i think if you if you could analyze the differences you'd really be getting a good index of leadership you know which set of leaders were better armed to take their company remote because there was nothing unforeseeable about this crisis and and by the way there were a hundred other different flavors of crisis that would have required the same solution i.e remote work and so this was something that a good competent leader should have anticipated and been building some muscles for and so it's like the old warren buffett quote like it until the tide goes out you don't know who's swimming naked like the tide just went out and and what can we learn about the quality of our leadership from how much of an emergency was sparked in the situation today's episode is sponsored by linode linode has 11 data centers worldwide including the newest data center in sydney australia with enterprise grade hardware s3 compatible storage options and the next generation network linode delivers the performance you expect at a price that you don't you can get started on linode with as little as five dollars a month and you can get talked about for a little bit more if you're interested in learning more about linode and how it's working and how it's working in the future pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl on GitHub. And they're actively updating this stuff. Go and check them out on GitHub as well. So Linode is a company of developers making tools for developers. Once again, you can get started with a $20 credit by heading over to linode.com slash developer T. It's linode.com slash developer T, all one word, and use the promo code developer T 2020. That's the word developer T and then 2020, the numbers at checkout to get that $20 worth of credit. Thanks again to Linode for sponsoring today's episode of developer T. Yeah, I saw a wonderful graph that made this very real. It was a it was a projection graph. And the projection of, of deaths, total deaths from from this virus. And, of course, it's a solemn, and very real thing to to imagine that. That, you know, just 10 or 15 days ago, the projections were half a million people. And this is a little animation. The graph stepped forward one day at a time. And the projections each day lowered. And so the all of the stories that we read the, you know, the New York Times article about flattening the curve, I think it was New York Times, that was visually made real in that moment. At the time, they were bringing these Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp that you know, and now it won't be. And that was such a, you know, bringing those numbers into a real picture for me was such a powerful moment. And it immediately made me think, believe it or not, of this upstream thinking. That to me was the same kind of heroic reward that I would have felt had somebody saved someone's life. In that moment, I had that same feeling that someone's life was saved. Yeah, I completely relate to that. I mean, it's like the thought experiment of the police chief that the policeman goes to that intersection and saves someone from getting in a wreck, but they don't know and the police officer doesn't know, but we see it in the data. And I think, I mean, maybe this is too meta since we're still in the middle of this crisis and people are still dying, but I don't know. I feel like this is a very powerful and unsolicited training exercise for the human species that we're going through right now, that we didn't ask for this, but then in a way, we need this if we're going to survive, that we need to understand the power of preparation and the significance of preparing for things that may or may not happen. On a smaller scale, I talked to a guy named John Koskinen, who was the Y2K czar. You know, back at the turn of the millennium. And, you know, there weren't many stories where Y2K was going to kill a lot of people, but people worried a ton about the collapse of the economy and power grids and so forth and so on. And I think it's got some similar dynamics to pandemic preparation because Koskinen told me he knew when Bill Clinton asked him to take this role that it was a total no-win position because if everything went to hell, he was going to be the most obvious guy to blame. And if everything went really smoothly, then people would just say, well, what was all that fuss about? There wasn't any problem here. This was just a big scam. You know, and I think the same thing could well happen with this. I mean, it wouldn't surprise me if three months from now, a lot of people have died, but far fewer than a lot of the predictions or forecasts suggested. And that people conclude from that, we just made a lot of fun. something that wasn't as bad as we thought, which is exactly the wrong conclusion, right? That the right conclusion was this was a massive threat that we responded to dramatically and at great cost. And as a result, we saved a lot of lives. But this is not a kind of logic that's taught in school, you know, and there's nothing particularly intuitive or natural about it. There's nothing in evolution that prepares us for these kinds of thought experiments, you know, there's no monkey sitting around analyzing the data of, you know, lion attacks avoided. It's just something that I feel like we have to grow into. Yeah, yeah, I think it is something that if you've had a chance to work with, you know, simulations or simulation theories or the ability to run, you know, algorithmic predictions, that kind of thing, you have a little bit more of a tangible grasp on statistics and how they work. And why they matter. But I don't think that the average person thinks in this particular way. And we see that all the time. We see that people use anecdote rather than data to make their point, for example, right? That's one of many, many things that kind of distorts the picture for a lot of people. My impression, by the way, is that the developers are much more natural upstream thinkers than other disciplines. Do you think that's accurate? It's a good question. And I think that, you know, I think it depends on which layer you're talking about. Because there are certain layers where developers feel that they're not at home. That it's not their problem. As you mentioned in your book, this isn't my problem, right? This is not in my realm. You know, I'm not the owner of this. Someone else should be responsible for this. And so the upstream, there's always more stream. And there's some source. And no one's responsible for the ocean, you know, or for the rain, right? At some point, you know, the responsibility, we draw a line. And I think developers, very often, they draw that line further downstream than perhaps they're capable of solving. And I think that's an organizational issue. But it's also a cultural issue amongst developers. How so? I think, so that's a good question. I think developers often feel that their solution. Should be hard solutions. When I say hard, I mean tangible, direct, measurable, metal solutions. And that anything that starts to dip into intangibility. Whether that's, you know, marketing, for example, is a really good example. It's a perfect example. Marketing or something that has to do with less measurable outcomes. Is where they start to draw. The line and say, I can't care about that. Because I don't have a way of measuring my success. And so if I can't measure it, then what am I shooting for? What am I optimizing for here? I'm just kind of optimizing to a black hole. I can't do that. And so instead, I'm going to work on things I know for a fact. I can succeed at. I think that's probably a healthy natural instinct is to be looking for data or other kind of measure. To use for navigational purposes. There's just a lot of devil in the details about which data you're using for that purpose. But virtually all of the upstream successes I talk about in the book. Profited from collecting the right information. And using it to inform, you know, the next set of steps that they would take. So it's a good instinct, I think. Yeah, agreed. And that means that, you know, developers have the opportunity. And I think it is an organizational risk. Right. Right. And it's a responsibility to push that, you know. And perhaps this is how we decide where we are on that stream map. You know, what point on the stream are we at? Well, we're at the point that we can measure. Anything that we can't or anything that is intangible, inaccessible to us. It's difficult to go up there. Right. It's costly to go up there. Because we don't know how far to go. Mm-hmm. Yeah. And just with the caution that we have. We also have to be very diligent about kicking the tires on whatever scheme of measurement we're using. Like there's an old story about AT&T that it wanted to boost the productivity of its developers. And so they decided they got really clever and started paying per line of code. Yes. And of course, how did developers respond by writing code in length? Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. off is it's almost inconceivable that we could succeed at preventing a problem before it happens without the use of smart measurement, but it has to be smart. Yeah, absolutely. Dan, I know we're coming up on time here, and I just want to ask you a very short couple of questions, kind of quick hit questions before we wrap, if that's okay. Sure. The first question, and I like to ask this of all my guests, what is something that you wish more people would ask you about? Good question. I think within the realm of this book, it might be what we were just talking about, the kind of devils in the details aspect of incentives and measurement schemes. There's a whole chapter in the book on just this problem. And I think that we've all heard this idea or this phrase of people gaming measures in various ways. And we often tell these stories in kind of a smirky way. Like, isn't it kind of funny that people get clever and play these tricks to try to earn better numbers or get their bonus or whatever? But I think it's much, much more serious than that. That I think, in fact, if we get the wrong measurement scheme in different places, and there's some pretty sobering tales in the book, including one from the New York Police Department of how measurement schemes can go wrong. If we get these things wrong, it can actually... It can actually doom the mission. It's not just a small thing. It's possibly the main thing. And the flip side of that is equally true, that if we get the measures right, especially if we can align short-term tangible measures that we can collect every week or every month with long-term goals in a productive way, that can be the secret that unlocks problem solving at a level not thought possible. Yeah, that's wonderful advice. And I think that developers who understand this, they know that this applies not only to the work that we do, but also the processes that we run, our teams, and certainly managerial work. And the engineering managers who are listening to this right now, they see immediately how this applies and how important it is to think really clearly about the measures that you're taking. We do this kind of work. And in tech, with OKRs, this is such a common way to do this, to kind of operationalize these measuring pieces and to try to balance the scales between two measures, for example. And it certainly is worth taking time to think about. I'll ask you one last question, and it's a short one. If you had 30 seconds to give developers of all backgrounds and experience levels a little bit of advice, what would you tell them? Oh, dear God. 30 seconds to glory. I tell you what, I'm going to mix it up and pull a concept out of a past book that I wrote with my brother, Chip, called Switch. And it's the notion of finding bright spots, which is to say, in times of change, our attention is usually drawn by what's not working. And we tend to analyze what's not working. We study projects that have gone wrong, or we do after-action reviews. But again, it's a concept. It's a concept. It's a concept. It's a concept. It's a concept. It's a concept. It's a concept. It's a concept. It's a concept. It's a concept. It's a concept. It's a concept. It's a concept. It's a concept. the best clue to a way forward is to study what is working, an unusually successful project or an unusually successful employee or an unusually profitable client. And that often by studying bright spots, we can find the secrets to our next moves. Man, that is good advice and certainly relevant for engineers. Dan, thank you so much for taking time to talk with me. What is the best way for people to find you? And also, where do you prefer to send people to pick up your new book Upstream? I am agnostic about book retailers. So whatever tickles your fancy, the important thing is to buy that book, of course. No, I'm kidding. If you want to learn more about the book, you can come to upstreambook.com. And there actually are a bunch of free resources at that site, a book club guide. And a podcast and a one-page framework summary and a bunch of goodies that you can get there for free. Excellent. Dan, thank you so much for your time. It has been an honor to talk with you. Hey, thanks. This has been a fun conversation. Another huge thank you to Dan Heath for joining me on today's episode. Dan has been one of the people that I've wanted to interview for a very long time on this episode. He didn't disappoint. I could imagine doing three or four of these interviews. I could imagine doing three or four of these interviews. And still learning something new from Dan in every part of the conversation. So I greatly appreciate Dan's time and joining me on this show. Thank you again to today's sponsor, Linode. Head over to linode.com slash developer T to get $20 worth of credit. And you get started with as little as $5 a month on Linode's Nanode plan, and you'll get root access to Linux server. Of course, you can go all the way up to having a dedicated CPU, a physical core for yourself. Head over to linode.com slash. Developer T to learn more. Today's episode was produced by Sarah Jackson. And the show is a part of the spec network. Spec.fm has a bunch of other podcasts that are relevant to you as a developer, especially if you are growing in your career. If you're interested in becoming a better developer, a better human, go and check it out. Head over to spec.fm. Thanks so much for listening to today's episode. My name is Jonathan Cottrell. And until next time, enjoy your tea. Thanks for listening to this episode of the Linode Podcast. I'm your host, Linode. I'll see you next time. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye. Bye.