Will is an engineering manager at Stripe, and he recently published a book titled, An elegant puzzle.
In today's episode ,we release part 1 of a three part interview with Will Larson. Will is an engineering manager at Stripe, and he recently published a book titled, An Elegant Puzzle: Systems of Engineering Management.
In this part 2, we dig into common patterns that help us create a fair evaluation system.
If you have questions about today's episode, want to start a conversation about today's topic or just want to let us know if you found this episode valuable I encourage you to join the conversation or start your own on our community platform Spectrum.chat/specfm/developer-tea
If you're enjoying the show and want to support the content head over to iTunes and leave a review! It helps other developers discover the show and keep us focused on what matters to you.
This is a daily challenge designed help you become more self-aware and be a better developer so you can have a positive impact on the people around you. Check it out and give it a try at https://www.teabreakchallenge.com/.
Our sponsor GitPrime published a free book - 20 Patterns to Watch for Engineering Teams - based data from thousands of enterprise engineering teams. It’s an excellent field guide to help debug your development with data.
Go to GitPrime.com/20Patterns to download the book and get a printed copy mailed to you - for free. Check it out at GitPrime.com/20Patterns.
Dealing with exceptions is kind of this like core core leadership skill, and particularly dealing with exceptions as you design process and figuring out how do you find it's sort of like architecting software. How do you find the common patterns? And how do you make them extensible to kind of handle like what reality brings you is, is a really powerful topic to think about, particularly when you're thinking about how do you also make this a fair and kind of just system, not just one that kind of gives the results that people wanted to have all the time. You just heard the voice of Will Larson. This is part two of my interview with Will. Will is an engineering manager at Stripe, and he recently published a book called An Elegant Puzzle. This is for engineering managers, but I recommend that engineers of all levels take a look at this book. There's lots of practical advice and tons of resources, as we discussed in the previous episode. My name is Jonathan Cottrell. You're listening to Developer Tea. Am I good? My goal on this show is to help driven developers like you find clarity, perspective, and purpose in your careers. And I think Will is going to help you do some of that in today's episode. So let's get straight into the interview with Will Larson. And you mentioned a just system. This is a concept that I believe you talk about in your book. And it's something that I wanted to bring up now because I think this is something that solves a lot of the questions. And I guess kind of conundrums that you might find, like, for example, in an engineering manager interview. You know, how do you establish a culture where the people feel, the people that are directly reporting to you, they feel like you're taking care of them, like you care about them as people, but also that you're demanding the quality level that is necessary for success. us. And I think that's a really hard thing for people to kind of wrap their minds around. And I think that this terminology of a just environment is exactly kind of the road to an answer to that. Can you describe what a just environment would look like? So when I think about kind of justice in process, which is kind of an interesting idea, I really think we want to make sure that people know how to accomplish things the right way, that the company has said, like, the way we want you to get up level is this, the way that we want you to be exposed to new roles and to apply to those new roles is this, the way that we want you to kind of have your compensation updated is this. And I think that's a really hard thing to do. And I think that's a really hard thing to do. And I think that's a really hard thing to do. We really want to make sure that people who do what we're asking them to are rewarded at least as well as people who don't do what we're asking them to. For example, if we're saying that there is a kind of a certain process to get your compensation adjusted each year, say there's some sort of automatic compensation adjustment system that happens to kind of bring you up to your current kind of whatever, like your title is, and the kind of pay band there, plus like an increase of a cost of living or something. We want to make sure that someone who is just following that process gets the same exact treatment as someone who's been advocating for themselves kind of every, every couple of months to get like a raise or whatnot. But also, we want to make sure that we're not putting some people in the position where they're just they got hired with a lot less money, so that they have to advocate for themselves consistently to try to get like a consistent result. But really, we want to make sure that no matter how people interact with the process, they get the same kind of treatment like approximately the same results. And to me, that's justice where we don't incentivize people to defect from the process. We don't incentivize people where they think kind of following the processes for suckers or whatnot. We actually want the best results to be for people who do what we ask them to do as we design these systems that they then live in. Yeah, I think that's important because it means that both sides have a responsibility. You as the company, the manager, you have a responsibility to know what you're looking for. And that's hard, but it's also useful. And if you can articulate what you're looking for, and somebody actually brings you what you're looking for, rather than creating this kind of mysterious environment where the picture in my mind is somebody kind of saying, well, you know, just impress us. That's a situation where... where it's very stressful to be a developer, but it's also unlikely to succeed, right? Because the way that somebody might perform in that environment is erratic. And it certainly doesn't match everyone's personality to go out of their way and find a way to, quote, impress their boss. That's a bad situation for everyone. One of the kind of like the open but slightly hidden secrets behind this is that there's a lot of evolution behind this idea of like consistency and fairness in companies and process is that people really know how to run fair processes. It's just hard. It just takes a lot of time. So consider two different examples. If you have like a really important project that you need someone to run, what you could do is you could kind of send out an email to the company, ask people to apply over the next week. You could make a clear rubric of how you'll select from the people that apply. You could go nudge people to kind of encourage them to apply people who might not think they're ready, but you think are ready. And then you could select someone, you could give feedback to everyone not selected explaining why they weren't selected. And then you could send an announcement saying who was selected to lead it. You could do all of that. Or you could just pick someone that that you think would do a great job and like move on to the next problem. And so one of the challenges is that like people want justice, but they don't necessarily want to like invest the time into doing it. And also, there are ways to make this cheaper, though. And so one of the things I've been thinking about is a lot of the processes that I've designed and seen designed are kind of just and fair, but but they're, they're really heavy. And so a lot of the work I've been doing in kind of management over the last year is trying to understand how do we find these systems that have the same properties of kind of fairness, and consistency, but also a lot less overhead to implement. And that that is when I think we get to this place, we're going to actually have kind of everything we want, where I think often when people aren't following these kind of more structured processes, it's just like a efficiency, like we need to get this done. Not not kind of a absence of desire to have consistency for the folks they work with. And can you share, you know, kind of what, what is an example of making it more efficient, and fair process? Is there something that you have in mind? That, like, like a process that you would actually refactor to be more efficient, but also still just? So going back to this example, like the selecting kind of leads for projects, like something we've done is we've moved towards pre materializing the selections, where one of the challenges is that we'd have like a project, like an emergency that comes up, we need someone to lead it. But we don't have time to do like a full process to like take a week and find someone that we need to do. And so something we've done is like, how do we like pre run these selection processes, where we can actually find the next person, a queue, right? Yeah, but exactly right. How can we do the work ahead of time instead of doing it in the moment? Yeah, that makes total sense. And it's, it's not so much about, you know, doing less, it's about finding the ways to kind of have on time delivery in that particular scenario. But I'm sure there are ways that we can, we can optimize for doing that. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. but because of some overlap or some particular scenario that we couldn't have predicted or somebody's uh you know particular tastes even the system doesn't necessarily cover everything and so i i wonder you know when you come up to these kind of messy situations where yes you you have this kind of idealistic way of dealing with problems but you know when you encounter these messy situations what skill sets do you think are important to fall back on when the model just isn't quite doing you know doing the work that you need it to do and instead you have to kind of do this manual uh kind of human work right it's almost hr work in a way um what are the what are those skills that you think are important for engineering managers to cultivate well when the model isn't working i think there there's kind of two different things to do there's kind of diagnosing the current situation and there's figuring out how to fix the model but i think you know that when something's going really wrong the the first thing i do is i try to talk individually with everyone involved and then i try to pull the group together and actually have that like kind of come come to truth team meeting where everyone has to like really kind of get their perspectives out and i i found as i've gotten better at running those meetings i've gotten better at running those meetings and i've gotten better at running meetings i found them extremely effective at getting kind of all the problems out on the table early on i think my comfort level with uh discomfort was not high enough for it but increasingly i think this have gotten really powerful so you know first listening asking second synthesizing and then really to me is like trying to understand the data going into the model and coming out in some cases like some human cases it's interesting like what does the data mean there's like just a few people involved but really trying to understand kind of the the principles that are driving conflict for example like if two teams are are really you know quote unquote misaligned and disagree about some problem it's almost always been the case for me that it's it's actually like there's like some sort of structural misalignment that has set them up for that not that there's some sort of like people who are fighting with each other or something like that but there's a lot of people who are actually like something real misaligned that that it really pays to try to understand the forces that are motivating the people to act this way versus just projecting that the people are kind of bad actors or something like that it's almost never the case that people are the bad actors it's almost always some structural thing driving them into weird behavior yeah i i agree with that completely it's a is it hanlon's razor or i believe it's yeah it's hanlon's razor not occam's but hanlon's razor essentially says that uh we should never attribute to malice what um what can be explained with pretty much any other explanation uh and and most of what humans are reacting to is some environmental factor um typically it's something that they don't necessarily understand or that they're trying to understand and and so i think i think what you're saying is is absolutely uh critical but it's also something that's hard to uh i guess see until you've fixed one of those situations until you've seen those principles at work and then you've said okay you know what is it exactly is this team uh aligned in a way you know on this particular project that they are responsible for too much for example maybe the the responsibilities are misaligned or maybe the expectations on one team are higher than the expectations on the other team and so i think those kinds of misalignments can be uh can drive behavior down the line that is you know one one two three four steps removed from the original problem to begin with there's a whole kind of like genre of kind of like the wrong goals led to the wrong kind of behavior um and i think this your your kind of comments fall directly into that i actually haven't seen too many cases in my career where there's like a lot of people who are like a story where someone had the wrong goal and they just did this totally wrong thing i've definitely solved the right problem in the wrong way where i sort of like regretted the approach longer term i think at uber we kind of spun up the service provisioning in a system in a way where people could just self-service and we ended up having essentially one service per engineer at one point and i think that was an unintended consequence of that approach but it was still the right problem and the right goal and it just didn't work out and i think that's the way i think about it just there were unexpected consequences down down the road from it i think that's i think what you're saying is is um critical to the kind of the systems thinking mindset right the hope is that with systems thinking and we won't go into too much detail here um because the book that you mentioned is is really a better way to to learn about it but the hope would be that you can identify you know once you've gone through the process of identifying the problem and you've gone through the process of identifying the problem and you've gone through the process of kind of modeling out a system of various interactions you can identify unexpected interactions unexpected results from you know a particular node in that it's kind of a graph problem typically when you look at a system a particular node in that sending a signal to another node you may not you know intuitively understand the relationship between two disparate parts of of that graph until you model it out and so i think that's a really good point and i think that's a really good point and i think that's what you're saying makes a lot of sense that you can look at these second and third order effects and uh if you try to solve the problem in one way those first order effects may look right right you may have part of the solution or you may have some of the effects that you want but these other unintended consequences certainly can happen too so one of the interesting things about systems thinking um is that oftentimes when you're thinking about solving a problem because like here's the goal um but systems thinking doesn't let you specify the goal. It just lets you specify the change, and then see the result. And so this is forces you to have like a really clear hypothesis about what you want to do. And instead of just thinking about the goal, it actually you have to change these levers trying to accomplish the goal. And I think that's really powerful. It's not just we will, you know, deliver five new products next year. It's like, how many engineers do we need to hire and when to actually deliver five new products next year? I think that kind of thoughtfulness by not letting you just specify the outcome, but actually specifying the action you will take that will create the outcome forces just like clearer thinking than any other approach I've really seen. Yeah, and there's there are tools to kind of model these things out, you know, to visualize what does that system look like when you when you put all the pieces together, you put all the interaction, you know, the interacting nodes or whatever you want to call it, where you say, okay, you know, we want to deliver five things. Well, let's just say, you know, we want to deliver five things. And then you can say, okay, let's put a node on here that's engineers, and then let's put, you know, unexpected absences, right. And so you would have those two things that feed into the system. And you may have, you know, 10 different things that you think are going to contribute or detract from you achieving that particular output at that point. And I think, you know, it's really interesting to watch as, you know, especially if you have some kind of automated way of running these things, it's interesting to see how, you know, an input in one particular area may have a feedback input to another area, right. And the feedback loop is the thing that makes systems thinking so interesting. Yeah, it's, that's exactly right. I think we often go into these things, thinking we understand like how they work, thinking we understand where the core constraint is. But so often, by actually taking real data of how things have worked, and trying to model that we learn just like, you know, how to do that. And so I think it's really interesting to see how that new things, for example, one of the things I've learned is that, like, typically, our hiring is constrained on engine manager hiring, not on our ability to hire kind of engineers. And so that has caused me to really rethink where my time goes when when we hire, and if I hadn't done that modeling, and then been wrong about it, I would have never kind of realized that was the right place for us to like put the majority of our time. Yeah, that makes total sense. And I do think there are a lot of these kind of unintuitive, things about, you know, the environment that we interact with, we think that if we, you know, directly work on hiring engineers in that particular example, that, okay, of course, you know, that's going to have the right return, but maybe there's a bigger lever. And I think that's kind of the, the, the, the key question is, what is the right lever for me to pull to get the things that I want? Today's episode is sponsored by Git Prime, We've been talking about engineering management in today's episode, and the previous episode of developer T. And you probably have noticed that the best engineering managers are excellent debuggers, they can see problems in code, but they can also see problems in teams. In a recent episode, we talked about the idea of pattern matching. And that is exactly what these great engineering managers are doing. They're looking for patterns, and what kind of patterns are they looking for? Well, that's what we're going to be talking about today. So, let's get started. That's what today's sponsor, GitPrime, helps you figure out. GitPrime has published a book outlining 20 patterns for effective engineering teams. You can get the book for free by going to gitprime.com slash 20 patterns. That's GitPrime, G-I-T-P-R-I-M-E dot com slash the numbers two zero patterns, 20 patterns. Thanks again to GitPrime for sponsoring today's episode of Developer Tea. I'd love to ask you a question, kind of switching gears here. What are some of the more unintuitive things? I guess it's similar in some ways, but more unintuitive things that you may have learned since becoming an engineering manager that you think other engineering managers, maybe they also have the same kind of misconception that you had. But now you see things a little bit differently. Are there some things that you wouldn't have expected to be true? But you found them to be true? Sure. This is a pretty large category. One thing that I thought was not true at all, which I now believe is true, is that I used to think that there were kind of. If I just had authority, like if I just had authority over this, I could accomplish anything. But really what I found is authority is pretty difficult tool. To wield at scale. So if you think about making a top down decision around some sort of org change or a top down decision around kind of changing the process to do something. It's so hard to do that the right way. You need to figure out the stakeholders. You need to figure out the impacted parties. You need to go talk to all of them. Build alignment. Get them engaged. Roll it out properly. Let everyone know through like structured communication. And that's kind of it's actually very hard to wield authority properly because it requires so much restraint. And I've really come to believe that if you to wield authority properly, you actually have very few kind of appropriate moves you're able to take in any given point. Like your flexibility is just like greatly constrained by all the constraints. Conversely, like I used to think, you know, operating without authority was very difficult. But what I really found is that. There's kind of a freedom of operating without authority, which is you don't have to follow all the steps. So a lot of these processes that I've experimented with about kind of how we do kind of interviewing people internally for roles that we're considering project selection. Cold sourcing. These are things we just ran as experiments internally to the team. And there is like none of the overhead of kind of doing it properly because it's just an experiment locally to the team. We're not asking anyone to buy into it. We're not asking people to do it our way. And I really think that to me has been kind of eye opening where in a lot of cases, the authority actually really slows you down because you have to be so thoughtful with it. And not having authority is, I think, sometimes liberating to actually move much, much faster. And that's something I would have never imagined was true until fairly recently. That is a very interesting kind of dichotomy of freedom versus kind of, I guess, responsibility is that the constrained. piece of that. And we see this in organizations, large organizations tend to make small teams. Innovation teams is what they're often called or innovation department that are less responsible for kind of the growth of the company or, you know, the success of the, for the stockholders. And sometimes they even make entire companies that operate inside of the company just to kind of shelter them. And so I think that this works both at the individual level and it works at the kind of the organizational level. You can watch this happening in very young startups. Young startups have the ability to pivot their entire business plan, right? Their entire strategy. And though that may not be easy on the kind of the emotional state of the people at that startup. Financially. It may be totally possible to do that. And, you know, it's kind of the big ship turning. You know, if you're light and agile, you know, lowercase a agile, then it's much easier to to make adjustments and to continue experimenting because the stakes are low. There's kind of an interesting juxtaposition here of kind of the innovators dilemma, which is one of the reason you have innovation teams in large companies is that the company. Business gets so valuable as you scale that it's irrational to do any investment outside of supporting that core business. And it makes it very hard to do new things. Conversely, I really don't like innovation teams because typically they create the social structure where like, why wouldn't you want to be doing the cool new thing with no production responsibilities? When if you have that option, like you should just do that. That seems like a lot easier. And so there's this interesting kind of friction between. This the business need and kind of the how do you design a process where you don't reward people for kind of bumming off and not doing the critical production work, the most essential work like run the current business. And then the business, if you only do that, you actually starve the business of like the future by kind of over focusing on today and figuring out how to balance those sorts of things, I think, is really some of the most interesting parts of kind of management and kind of. Just thinking. In this domain overall. Yeah, it makes total sense. I think it makes sense to have your department, your engineering department, keep a pulse on whatever is keeping the business alive. And, you know, artificially shielding a group of individuals from that is can be can can so kind of this bad discord amongst the people who aren't in that team versus the people who are. It's like the. You know, what made what made you guys so special. Right. And that's that's probably not a healthy culture, but it also could could mean that, you know, the operational mindset is entirely different. And so moving whatever that innovative thing is into the mainline business may be a huge task if it wasn't kind of integrated to begin with. That is definitely one of the kind of the most painful anti patterns, which is the innovation team. That does a handoff to the maintenance team to kind of run the new thing and then moves on to their their new kind of newest shiny thing. I think, you know, if you look at Stripe, we've we've started to launch more and more different product lines as we get older. And I think one of the things that we're trying to do is we're trying to layer the products on top of each other, not kind of strictly compete and not strictly kind of extend the existing functions. But actually build things that are built on top of our existing platforms. So I do think there's ways that you can be kind of thoughtful about the organizational and business design to kind of avoid some of these kind of downsides, but also allowing you to innovate, move quickly and kind of not get kind of stagnant in how your company grows. Yeah, that makes sense. I'd love to switch gears for a moment here again and ask you kind of from a high level as as people are as engineers. Are growing in their careers. They're going to face the moment where they have to kind of make a decision. Are they going to continue being an engineer for the rest of their career or are they going to kind of develop these management skills? And I was talking with Matt Klein recently who created Envoy's at Lyft as a long career as an IC and he wants to stay an individual contributor. So I'd love to know. You know, what advice do you have for someone who is facing this conundrum of, you know, do I do I go into this different kind of path into management or do I continue being an engineer? Do you have advice for people who are who are facing that decision? So one of the kind of perverse things of Silicon Valley kind of engineering in particular, but as I think also common across kind of engineering in 2019 at large is I think we're we tend to like have this like default assumption. That we're going to be in the industry for like 10 years or 15 years. That is kind of this really short term thing. So, for example, you're you're you're five, six, seven years into your career as a manager or as an engineer. Let's say you're a senior engineer at a company and someone comes to you and say, hey, do you want to do you want to try being a manager? I think people are often like, well, if I try being a manager, it might not go well and it's going to take me six months to know it's not going well. And then it's going to take me three months to re-ramp. And I'm going to lose nine months on my kind of career track to getting promoted to staff engineer or something like that. And I totally understand like why kind of people get into that mentality. But the reality is, like, if you're going to be in this industry for 30, 35, 40 years, losing nine months to try out a new role isn't isn't that big of a deal. Like you're you're still getting paid. You're just like haven't gotten promoted at the same pace you were expecting to. So my first thought is that I think there is a sense that experimentation is far more expensive than we think it is. And, you know, this isn't true for everyone. And so if you are supporting several generations of your family off your income, you have a lot less risk tolerance. If you have younger kids and you're supporting them, you have a lot less risk tolerance. And so it's not this is not universal advice. Like everyone has to understand kind of their risk profile. But in general, I found that people. Really ought to try management if they're interested in it. They don't like it. You pick up a different perspective. I think it's really powerful perspective. And then you can just bounce back off. If there's almost no permanent decisions you can make. I think people frame things permanently sometimes. But but I think far more people go into management and then kind of step out of it after a couple of years. And I think people consider I think it's pretty normal to kind of go through that path. And those folks, I think in the. Moment, they feel like they've lost something. But so much of being an effective staff engineer is the same sorts of skills that you have to build to be an effective manager. So I really think there is an accelerant here. It's kind of like speed slowing down to speed up if you want to go back to the IC path. But I really, I really don't think there's harm. And I think there's a lot of upside if you give it a try. Yeah, I totally agree with this idea that we have this kind of mental picture of our careers. That, you know, the next six months are incredibly important to the long term reality. And that's not just to discount the importance of what may happen in six months, but rather to say that, you know, the amount of change that you're going to go through your career or throughout your career is it's going to continue to grow. And it's you're going to continue facing these new situations. You know, one of the things that I said early on in the life of this podcast is as developers are trying to, especially very young engineers, if you're trying to decide, for example, what language should you learn? And your head is spinning because there's so many new things that are coming out left and right. And you're trying to decide, OK, you know, what should I do? And you start learning one language and then you hop up, hop over to another one and try to learn a language a week almost. And the truth is, you're probably better off. If you just went ahead and committed to one language for six months. And a lot of people who are listening to this right now, they already have a little bit of anxiety thinking about committing to a to a language for six months. Right. But the truth is, you know. If you if you look at the grand scheme of your career very quickly, that six months that you invested to develop a much deeper understanding than you would if you were hopping around to every other language and just learning. Hello. World. Six months is going to return very quickly and you're much less likely to to fall behind than you may immediately think you are. And I think the same principle applies to to trying out a position as a manager. So much career advice is kind of just here is this incantation that I use to get into this cool role. If you to pursue this incantation, you will also get in a cool role. And that could be, you know, learning a language every week for for 10 weeks. That could be, you know, writing an engineering blog. That could be being a conference speaker. That could be like almost almost anything. There are more ways to kind of be successful in your career than I think any one person can can list. There's just so many different paths to follow that can kind of get you there. And I think. It's easy. I've certainly been very anxious at times in my career about how do I get to the next thing? The right thing. But but there's so many different paths to follow that will get you there that that honestly, I think just as you said, focusing, doing some some good work, building some depth. Just do that over and over and you'll you'll get someone or really, really, really good. Thank you so much to Will for joining me on today's episode and the last episode and the next episode of Developer Tea. This is a three part interview with Will Larson. And of course, thank you to today's sponsor, Get Prime. Head over to Get Prime dot com slash two zero patterns. That's Get Prime dot com slash 20 patterns to read the free book about the 20 patterns you should be looking for on your engineering team. Thank you so much for listening to today's episode. If you're listening to this show, that means that you care about your career. You are looking to level up as an engineer and Spec.FM was built specifically for you. Spec.FM. Spec is a network of podcasts, including this one that is totally focused on helping you as an engineer level up in your career. Go and check it out. Spec.FM. Thank you to today's producer, Sarah Jackson. My name is Jonathan Cottrell. And until next time, enjoy your tea. See you soon.