Will is an engineering manager at Stripe, and he recently published a book titled, An elegant puzzle.
In today's episode ,we release part 1 of a three part interview with Will Larson. Will is an engineering manager at Stripe, and he recently published a book titled, An Elegant Puzzle: Systems of Engineering Management.
In this part 1, we dig into why Will chose that title and about his developer journey.
If you have questions about today's episode, want to start a conversation about today's topic or just want to let us know if you found this episode valuable I encourage you to join the conversation or start your own on our community platform Spectrum.chat/specfm/developer-tea
If you're enjoying the show and want to support the content head over to iTunes and leave a review! It helps other developers discover the show and keep us focused on what matters to you.
This is a daily challenge designed help you become more self-aware and be a better developer so you can have a positive impact on the people around you. Check it out and give it a try at https://www.teabreakchallenge.com/.
I want to take a moment at the beginning of this episode to thank everyone who listens to this show sincerely. We just recently reached 700 episodes this past week, and it's been such a wonderful experience to create this podcast for four and a half years now, and we have no plans of stopping because the purpose of this show is inspiring every single day for me. And I just wanted to take a moment for thanking you for making this possible. But I also want to thank all of the wonderful guests, including today's guest, Will Larson. Will is an engineering manager at Stripe, and he recently published a book through Stripe Press called An Elegant Puzzle. This book is chock full of fantastic information, lots of practical resources and guiding principles, for engineering managers. So I encourage you to pick up that book if you're interested in the subject. My name is Jonathan Cottrell. You're listening to Developer Tea, and my goal on this show is to help driven developers find clarity, perspective, and purpose in their careers. And I believe Will has similar goals in his day job and as an author and as a guest on the show. Let's get straight into the interview with Will Larson. Will, welcome to the show. Thank you so much. Thank you so much for having me. I'm very excited to talk to you, not only because of your experience at Stripe, but also because you just recently released a book. And I believe that this book is really going to kind of go into the classics pretty quickly for software engineer managers. And I'd love to talk to you about the book. But first, I want to ask you about the title of the book. The title is An Elegant Puzzle. Can you kind of explain where this title came from? So titling books is kind of an interesting process where you kind of basically the working title for a long time was kind of this like systems of engineering management or engineering management systems. I knew systems was an important part, and it's a book about engineering management. So maybe we should acknowledge that. But really, really kind of knew it was a bad book. It was a bad working title, but it wasn't quite sure what to talk about. So as we were starting to write up the promotional materials around it, like what did we want to tell people about when we talked about the book? One of the things that really came out was this idea that engineering management really is about solving these interesting puzzles and kind of figuring out these interesting patterns and how to use these patterns to solve the day to day problems that we run into. And kind of as we started riffing on that idea, we, um, someone actually just pulled it out. Uh, this phrase out of, um, this, this longer thing I had written kind of about promoting the book. And it's like, why don't we just call it this? And it really stuck. I think it was one of those moments where it was like, this is what we've been missing. This is like this, this awesome thing. And it was just so much better than kind of the previous titles that we've been thinking about that. Um, we just kind of pivoted and went with it from there. And I'm, it's it, uh, the title along with the cover are seemingly like you wouldn't imagine. Those are the most important parts of a book, but really when I think about what makes me so happy about the book, I think that the cover image and the title are really right up there. I agree. I have a copy of the book and the kind of the texture of it and the weight of it, um, really the whole kind of the quality, the feel of the cover. Of course, you're not supposed to judge a book by its cover. Um, but this one has a pretty good, pretty good face. You know, the, the, that initial impression, um, and you know, it should be, it should be noted that this was published through Stripe and that's where you work every day as an engineering manager. Is that correct? Yeah. So this was the, I think the fourth or fifth Stripe Press book, and it was the first one to be, be written by, by, uh, Stripe. So it was pretty, pretty exciting both to be the fifth book, uh, this awesome kind of new publisher, but, but also, um, to get the opportunity to be the first Stripe to write one was pretty, pretty special as well. And when you say the first Stripe, you mean the first person who that's, that's the, the internal, uh, noun for someone who works at Stripe, right? Yeah, exactly that. Cool. So, um, the, the book is full of this very practical knowledge and it, I wrote down a note here before I forgot, uh, this book, it really reminds me quite a bit of, uh, the personal MBA. I don't know if you've read it. It's, uh, by Josh Kaufman. Excellent book. And it does a very similar, very similar thing. It goes through and provides you some very practical knowledge about very specific things, but then it does, you do this thing where you kind of zoom back and you say, okay, I'm giving you kind of the, the, the practical knowledge, but there's a lot more here. There's a lot more that you can dive into a lot more discussion to be had. And here's the references and the back of the book, you just have this huge glossary and, uh, uh, index of references. And so I wanted to talk to you about that for a moment. Have you actually read all of this stuff? That's an amazing amount of material, number one and number two. Um, you know, for people who are interested in kind of learning these kinds of systems, do you have maybe a list, a top three of things that you would recommend, uh, for people to, to read once they've already, of course, once they've read your book, right? What are the next two or three books that you would recommend for someone who's interested in, uh, a career as an engineering manager? So I, I, I once, you know, a while ago got scolded for bringing up books that I had read too frequently. Uh, it was, I was, you know, this, this book kind of raises this important point about, about something we're discussing now. And I think the, the CEO at the time was not amused, like stop bringing up books. Uh, we don't, we don't need that sort of stuff around here. Okay. So, so, so, so, so, so, so, so, so, so, so, so, so, so, uh, so, so yeah, I actually have read all of those books, all those papers. Um, and really, I mean, for me, like reading has been like a lifelong passion and, and writing as, as well. But, but yeah, I, so much of what I've learned and so much of what I think many managers learn is, is from reading these books. If you're at a small company, there might be one other engineering manager. There might be no other engineering managers. Where do you go to like kind of learn about the practice? And I really, I really think these books that, that, the handful I've selected and the kind of appendices I think are ones I've really loved, but I think the ability to learn when like the specific environment you're in is not rich with experiences. I think it's just, just this special property of kind of books and written material. Um, the number one book I recommend to everyone that I talked to is, um, think thinking and systems of primer by Donella Meadows. It's just so, so good. It's just like an entire new way to think, um, that I, I've really just found powerful. Uh, another one I would think kind of a classic, but I think people where is a, um, by I think DeMarco and Lister is a, another kind of, kind of seminal work, but just such a powerful thoughtfulness that has really just transformed. I think so many managers, that's like the only book they've ever read about engineering management. And it has a really special place in my heart as a result. Um, and then another more recent one that I've really thought really highly of is just exceptional. Accelerate. And that's by, um, Nicole Forsgren, um, I think Jean Kim. And there, uh, is a third author who's I'm slipping from, from my head, but just again, a great data driven look at how management and practices actually work. Those, those might be my three top ones to start with. Yeah. I, those, uh, I've, I've, I've read people where, and I've read summary of the thinking and systems. Um, I have not actually dove in and read the book. It's sitting on my, on my shelf. Of course, I have way more books than I have actually read because, uh, that's, that's just a bad habit of mine. I suppose I, I find a book and I follow the rule that, uh, if there's ever a book that I'm considering buying, go ahead and buy it. That's, it's probably not a bad investment, right? We, we have like a fixed size, uh, bookshelf that my wife and I have to like one book in one book out. So we, we've sort of fought that because otherwise we would have literally nowhere to, to, to put things. Yeah. We have, uh, we do not have such a rule in my house. So, um, whatever flat surfaces are available, there may be a book on that, on that flat surface at any given point in time. So I think, uh, as, as a moment of, uh, I guess advice for people who are like, man, I really wish that I could read as much as, as these people read. Uh, one of the things that was really liberating for me, and maybe you have a similar experience that you can share, uh, was the idea that, you know, you don't have to read every last page of a book for that book to be valuable to you. Uh, number one. And number two, you also don't have to put a book down forever. You can go back and read it in different portions, especially with management books. Uh, you can open up a book and it doesn't have to be read like a narrative. Uh, like we were taught to read, you know, in, in grade school, you can use this book more like a reference, a source that you can jump through, jump around, uh, you know, pick up and drop different places that you want to pick up and drop. And I think that was really liberating for me, uh, where before I felt like the only way that I can truly say that I've read a book is if I've, you know, devoured every single word in the book. But I think everybody has different styles. How do you, you know, have you found that that's the, that you found like a similar approach to reading or do you read every single book and the introduction and the table of contents? I think what you just said, it resonates a lot with me. I think, you know, one of the challenges, like we're typically brought up as readers, as readers of fiction. And that's kind of the first thing we start reading. And, and fiction is, fiction's not that much fun. If you skip around, fiction's not that much fun. If you skip parts, you like miss the plot. Um, there's like beauty in the description. Um, and, and I personally, no one ever told me like when you're reading this other sorts of stuff, you just skip to the content. But that is, um, exactly as you said, something I've learned, over the last five or six years was for books. I don't really like, but I want the ideas. Just skip, skip to the ideas. It's usually pretty easy to find them. These books are pretty structured. And yeah, I, there's a lot of books now that I have read that I have really skimmed through finding the, the, the nuggets. And, you know, 10 years ago, I would have never, never even imagined that. But I think there's a book called like how to read a book or something, which, which I've been told, uh, is, is quite, quite useful for learning how to, how to do this. Yeah. It's, it's really important to, to be able to do this. I mean, use whatever resources are available, right? I mean, um, you know, if you get 10% of the value out of a book, that's better than 0%. And I think that's, that's something that a lot of people, you know, they, they never pick up a book because they feel like it's this daunting task, uh, to actually get through it. But there's, there's so many ways to, to get value out of that. And just like building on that real quickly, like one of the, the realities of the publishing businesses, and one of the reasons that I really, I love Stripe Press is that most other publishers have like target lengths where you need the book to be like 250, 300 pages. And then you, you, you fill it, you, you, you get the, you, you get the, you hit the metric, right? But it doesn't necessarily translate to like a great book or a really readable book. And so I think even the authors aren't necessarily putting out the books they want to. Um, and one of the things I really appreciated about my book is that I felt initially like it was a little bit short. I think it's 250 pages, pages approximately. And it felt a little bit short when I held it, but like, no, this is fine. Like if these are the ideas and this is the amount of space you need, let's do that. And that was something really special as well. Yeah. I think, uh, your book specifically is another one of those books that it's not a book that you read to get all the ideas and then put down and, you know, put it in the archive. your, this book is much more like an index of information. And, um, various systems, various kind of structures, ways of thinking that can be reused as, as kind of like templates or models in the future. And I, I don't see it as, you know, uh, the same kind of thing as, as reading through, you know, a psychology or self-help book. It's, it's much more, um, it's more, more like, and this is going to sound wrong at first, but it's more like a recipe book in the sense that, you know, you can reuse these pieces of content, and it's not something that you're going to hold in your mind per se. Um, but it is something that is seated well in, uh, kind of, uh, uh, research other people's research, your own experiential, uh, um, like your path as an engineering manager at, at Stripe. And so, yeah, I don't, I don't see myself taking this book and saying, okay, yes, I have devoured the contents of this book and now I can move on. It's much more of a, you know, this is going to sit on the shelf as a reference. So something you said earlier is like when you're talking about it being similar to the personal MBA is, I think really what you said just now kind of resonates, which is there's a recipe and this recipe is basically going to work for you. Most of the time, it's not perfect. It's not like the tastiest reorg you've ever, uh, eaten, but it's like, it's a, it's a passable reorg. Um, but then like, then kind of segwaying, it's like, and here's the way to think about this as well. And I think, you know, sometimes, sometimes you want the kind of the thinking and the approach behind this and like all of the details. Sometimes you just need like a recipe to like run. And I think trying to give both, that's always a goal for me and kind of the different things that I write on, on my blog and also in this book. Yeah, I totally agree with that approach. I think it's important to, you know, understand kind of the system or the systematic way of arriving at a particular output. And then also saying here is an output, right? It's, it's kind of the both sides of, of that, of the system, not just, you know, theoretically, this is what we should do, but also here is an expression of how that would play out. Cool. So I, I'd love to ask you on that point, you know, a lot of these systems that you lay out, they're explicit systems and they're explicit, like even numbers, right? So for example, one of the things that you mentioned in the book is, you know, team sizing. And, you know, this is kind of a fundamental concept when you're building an, an engineering organization, the size of the team and how many people a given person can manage. And all of this is certainly from some of your experience, but I'd love to know, you know, what ratio would you say this is your experience versus this is kind of theoretically, or at least, you know, backed by some kind of research. This is the way that you should go. How much, how much would you say you've, you've kind of had in your career versus, what you believe based on, you know, some, some given research. There's not, there's not a ton of research on these topics. And that's one of the reasons why I think accelerate is such a great book because it actually is research driven, but also people where, and slack are, are older books that had some research component in them as well. And I think that is part of the reason like these three books like resonate with me so much. And for, for my, for my writing, I think it's generally been my experience that it's hard to find kind of conclusive information about like what size team should you have, but, but you do have, I think in these fast growing companies. And when I was at Uber was growing, you know, four, four, 400% year over year, or when I was there for about a little bit over two years, and I grew from 200 to 2000 engineers in that period. And one of the really neat things about this hyper growth is that it's kind of this, like Petri dish of org experimentation, where you get to see like so many different versions of so many different things. And then one of the things I've loved about coming to Stripe and being here to the past three years is that it's growing quickly, but it's growing a little bit more slowly where I get to kind of recreate these different organizational experiments, but more meticulously and more thoughtfully. And I know kind of what's coming down the pipe. So both kind of working at these like fast growing companies, but then also reaching out across the industry, to kind of folks and understanding how they work at their, their other companies. It was like benchmarking, but just kind of understanding how do you like 10 similar companies like approach the same problem. I use this a lot for kind of as a general problem solving technique, but also specifically in this case, um, this is something I spoke with a bunch of folks, different companies as well. So I, I unfortunately don't think there's a whole lot of great data, but kind of this, but on the other hand, I think arithmetic is really powerful in these cases. Um, and I think Andy Grove was talking about how to size, how he sizes teams. And I think he comes to eight, um, or, or he might've come to five, but he had just like some very simple kind of rule of thumb, which is like you spend half a day per person, so you can't have more than 10. It was basically how he did it. And I think these like rules of thumbs based on observing like the actual kind of constraints of, of your time tend to be surprisingly powerful, even though initially they seem too simple. I've seen them outperform kind of any other approach. That I, that I've seen so far. Yeah, that makes total sense. I think, um, you know, sometimes we, we try to complicate these things and it's easy to think that, uh, you know, our, our particular system is going to work just because we saw it work elsewhere. But sometimes we just need to look at kind of the, the size of the box, right? Um, you can, you can do your own kind of principles based understanding of, of the situation that you're in. And derive some of the answers for yourself. And I think the book is really good at explaining that. Um, and a lot of the topics that you cover are, you know, it's, it's like, oh yeah, well, of course, of course it's that way. Um, and not like, of course it's that way and we all talk about it, but more like, of course it's that way. Why didn't, why don't we talk about it more in this, in this frame? Something I, I have like a kind of a stint, a year, two or three years ago about kind of reality based decision making, which is kind of this like thing I thought about kind of constantly, which is that it's strange to, to, to your point, there's all these things we know are true that we don't act on. And I think the most obvious example is like the, um, kind of open office floor plan. There's a lot of research showing that this is not in fact like a good way to make productive teams, but we, we keep, we keep perpetuating it and we keep explaining it with the same explanations, you know, like more collaboration, more communication, even as the data comes in kind of disproving it. Um, we, we still, we still keep focusing on it. Um, a book that I read recently, which is not a technical book. Um, but I read the two income trap by, um, Elizabeth Warren and her daughter. And they do a lot of work around research about kind of explanations for certain types of kind of financial hardship and kind of like how the narratives that we hear in kind of like written articles, um, and then the actual science refuting those narratives. And, and I think what was so powerful about her book was just how much many of these narratives are things that I've believed historically, and then seeing the data dis dis proving them. And I think it's, it's both powerful to know that you are wrong about a lot of your beliefs. Like a lot of your beliefs are actually not grounded. They just are reasonable. And so they trick you with the reasonableness. Um, but then there's other things that we actually do know and then just don't act on. Um, and thinking about that, I think is really important as we try to make these decisions that like impact kind of the, the lives of everyone we work with. Yeah. I heard a very interesting discussion on a similar topic recently. I think it was on the hidden brain, the podcast hidden brain by NPR. And they talked about, or it may have been on freakingomics. One or the other, they talked about the, um, the difference in the way that humans and dogs, domesticated dogs, uh, interact with our environment in the, the specific difference that they pointed out. Of course, there's a lot, but the one they pointed out was that humans are, uh, very much so kind of copycats. Um, and we, we imitate each other and we imitate, you know, what someone tells us to do. We're likely to kind of listen to them and we're likely to do things the way that we see other successful people doing them. And so it, it absolutely makes sense that we would have a reality around us that we ignore, right on the one hand. And then on the other hand, we're willing to take this kind of conventional wisdom or, uh, uh, uh, these narratives that we've heard over and over and over and assume that they're representative of some reality. And it's kind of this, this double-edged sword where we aren't really observing the reality around us. And on the other hand, we are also accepting some false reality and this can perpetuate into really bad decisions. Um, you know, what you made, what you said about sense-making or about, uh, uh, you know, reasonableness fooling us. Uh, this is really, really critical for, especially I think managers to understand because a lot of our kind of retrospective practices, we can look at, you know, what went wrong. Well, if we can explain what went wrong, then that explanation, as long as it seems like it falls together and all the pieces of the puzzle fit, we're apt to believe it. Um, and even if the explanation is actually, uh, well, some set of random events occurred that we can't really explain. And that's why we're, at where we are, that may be the real explanation, but we're not very apt to believe that one. We're much more apt to believe the one that supposedly makes more sense. So one of the things I've learned the most about over the last year, um, is, is kind of incident programs. So kind of these programs where you have incidents in your company, you kind of remediate the incident and you write up like an after action report or like a root cause analysis or whatnot. And then you kind of review them in some sort of group together. Um, and I think, um, I really learned a lot from some of my coworkers, Talina and Davin in particular, who have been just like open my eyes to how these work, but how I, how I used to think these worked was that you brought in a report. Um, you talk to people about it and helped them improve kind of the quality of their analysis and maybe the quality of the remediations and you were done. But what I've learned from, from working with them is that the most important thing these, these incident programs do is they create, um, pristine data about what's actually happening. And it's this metadata, which then allows you to see trends that you can actually believe in. And then it's the discussion of these trends and the metadata, and you start solving the trends, not just solving the incidents. It allows you to actually have the data you need to actually have that right conversation versus kind of the, the, um, immediate conversation reacting to any given one, which I never even realized was, was the, the goal of these programs until fairly recently. Yeah. That, that would be, I think an invaluable tool set, um, for developers to have. And I think, uh, developers, engineers, managers, uh, you know, all of us are doing this kind of evaluation and, um, you know, this, this is applicable outside of engineering. Of course, uh, anybody who is doing work and trying to improve on it, trying to iterate on it and, and kind of improve their working processes, uh, makes total sense to be able to, to evaluate trends, um, rather than, you know, one of the things, kind of an anti-pattern in engineering management is the idea that, you know, a given incident is representative of some larger reality, right? That, uh, and the, the trouble is that humans are much better at remembering individual incidents, especially if there's something, you know, extraordinary about that incident. And so it's easy for a single failure to stick out above all of your successes. Right. Um, and so, you know, this, this visibility of, uh, of a particular moment in time can be really, uh, a bad metric to use. Well, it is a bad metric to use to evaluate somebody's overall performance, certainly, but it's also kind of the easiest one. It's we, we use these signals as, as our brains are able to process the world. We don't really process it in total. We process it one piece at a time and we try to take the most salient pieces, uh, and that just so happens that the more extra extraordinary ones are the ones that we're likely to take. No, absolutely. And I think it's, um, you know, like flame graphs for debugging kind of performance and production. It's like, if you take any one slice, you don't necessarily get something good, but as you take many different slices, they slowly start coming together. You get like an accurate sense of what's actually happening. But, but if you only, if you only take a couple of samples, then it's almost always, you're going to like focus on kind of, kind of fixing the wrong problem and being really careful to be, you know, looking at the distribution, not just the singular event is one of the most important things that I sort of have done to improve the quality of my, my thinking and decision making. So we allow these, um, these extraordinary events to kind of act as landmarks, uh, for, for our evaluation of what, uh, our, our direct reports have done. Even what we have done, we can look back, we can see, you know, the, the moments of failure, uh, very easily. But sometimes we don't see those consistent moments of, of success. I think something interesting here is there's kind of like two different scenarios that are interesting to think about for how you handle kind of outliers. And one of them is kind of the sort of thing you're talking about, like a failure and how we think about, oh, how do we avoid over indexing on the rare failure? And then conversely, when we think about process, we actually want process when we design like a promotion process or redesign a hiring process. We actually want those processes to handle all the exceptions in like a really intentional way. Maybe make those exceptions not be exceptions, but actually handled in the norms of the process. I think getting better at making decisions of kind of all of these different varieties is definitely one of the hardest parts as being a leader of any sort and kind of improving your toolkit. When do I need a distribution? When? What is significance? What does significance actually mean? Like how many data points do I need to actually understand if this is improving or getting worse? Like that kind of fundamentals of your statistics. And then thinking about how with process, it's not about statistics. It's about like every single case getting handled thoughtfully. Like all of these pieces, I think are really interesting pieces of the toolkit that you have to kind of bring to it. Yeah, I don't think a good prescription would be to just ignore those exceptions. And I think that, you know, finding that proper, you know, proper balance is incredibly important. And process is a way to kind of enforce a balance there. And good process. I think I was talking to someone kind of recently about good process. And, you know, one of the things that makes good process good is kind of the consistency of application. But one of the things that makes good process or bad process bad is kind of the needless urge to kind of force everyone with the same exact system. And, you know, I think that's one of the things that makes good process good. And I think that's one of the things that makes good process good. And so I really do think there's kind of dealing with exceptions is kind of this like core core leadership skill and particularly dealing with exceptions as you design process and figuring out how do you find it sort of like architecting software. How do you find the common patterns? Then how do you make them extensible to kind of handle like what reality brings you is it is a really powerful topic to think about, particularly when you're thinking about how do you also make this a fair and kind of, just system, not just one that kind of gives the results that people wanted to have all the time. Thank you so much to Will Larson for joining me on today's episode of Developer Tea. If you don't want to miss out on the next two parts, this is a three part interview with Will. We had so much ground to cover. If you don't want to miss out on those two parts, I encourage you to subscribe and whatever podcasting app you're currently using. And here's why I think you should subscribe. You hear all the time that if you, put your investments, your savings on an automated schedule, you're much more likely to save more than if you were to try to remember to do that yourself. Now, the show isn't necessarily a financial investment in your career, but I do believe that Developer Tea, other shows on the spec network and the other wonderful podcasts that are in the kind of developer podcast ecosystem, that those are investments into your career. Subscribing doesn't mean that you have to listen to every episode. And in fact, I encourage you to pick the ones that you think are most relevant and spend your time with this podcast strategically. But I do believe that spending regular time thinking about these subjects is important to every developer's career. By the way, we wouldn't be able to make this podcast happen without spec. Spec is a content network that is built specifically for designers and developers who want to level up in their careers. Go and check it out. Spec.fm. Thank you to today's producer, Sarah Jackson. My name is Jonathan Cottrell. And until next time, enjoy your tea.