Robby is the co-founder and CEO of Planet Argon and the original creator of OH-MY-ZSH. In this part 1 of the two part interview with Robby we focus on Robby's background and importance of maintainability on your team.
Thank you to long time sponsor and friends of the show Linode for sponsoring today's episode!
Simplify your cloud infrastructure with Linode’s Linux virtual machines and develop, deploy, and scale your modern applications faster and easier.
Listeners of Developer Tea can now enjoy $100 in free credit! You can find all the details at linode.com/developertea.
If you enjoyed this episode and would like me to discuss a question that you have on the show, drop it over at: developertea.com.Â
If you're enjoying the show and want to support the content head over to iTunes and leave a review! It helps other developers discover the show and keep us focused on what matters to you.
Hey everyone, today's episode is about maintainability. It is an interview with Robbie Russell. My name is Jonathan Cottrell, you're listening to Developer Tea, and my goal on this show is to help driven developers like you find clarity, perspective, and purpose in their careers. Robbie is going to help you do some of those things in today's episode, and we talk a lot about maintainability. This is something that Robbie and his team really invest a lot in. This is all that they do, essentially. I hope you enjoy this interview with Robbie Russell. Robbie is the co-founder and CEO of Planet Argonne. He is also the original creator of Oh My Z Shell, which is open source. There are many creators, but Robbie is the one that started that project. Let's get straight into my interview with Robbie Russell. Robbie, welcome to Developer Tea. Thank you so much for having me. The thank you goes to you for having me on your show previously. You want to tell folks about that show? Really briefly, before we dive into some of the topics that we've got today? Yeah, sure. So I'm the host of Maintainable, which is a software development podcast where we talk with people like Jonathan about how to overcome the problems often associated to technical debt and legacy code in older codebases. And so it's an interview style format. And yeah, so you can find that on maintainable.fm. Maintainable.fm. So I'm excited to talk with you today because you've got this. You've got a couple of topics that you mentioned in advance to me that you are interested in talking about. And it just so happens that some of these topics are things that I'm personally experiencing or have experienced multiple times in my own career. And I'm certain that other people who are listening to the show are either currently or will one day or have in the past experienced it. And I think you have a unique insight on this. But for people who don't know about your work, can you kind of back up and provide just a little bit of insight of what your day to day looks like? Yeah. So I am the CEO of a company called Planet Argonne. And we've been around for a little over 18 years. And we primarily are working these days for the past 15, 16 years or so with Ruby on Rails. And so we're a consultancy. So we come in and help organizations that have already built a Ruby on Rails application or have it as part of their infrastructure. And we come in in a consultative supporting role where we're either helping them with overdue upgrades or helping them work on some performance updates, security issues. Or adding and building and extending new features in their applications. Or there's another or more often, we're also a company that will take over those applications for companies where they don't necessarily have their own internal team. Or maybe they had one or two developers for a really long time that have since left to go join a larger team. And now they're like, well, we don't know that we're the best fit to manage software developers full time in our company. But we can work with a company like Planet Argonne where they have more than a few people that can provide a more well-rounded team from design and development. And back end type of approach to that. And so then we become the team that kind of manages that project for a number of years going forward. So primarily we're consultants. Yeah, that makes sense. And is such a kind of a unique position to be in. There's so much that people like me who are not in a consultancy role can learn from you. And I want to talk about some of those things today. Also, just a quick mention or shout out to probably a project that many people who are listening to this podcast are already using. Oh, my Z. Oh, my ZSH. Oh, my Z shell. Whatever you call it. What do you call it officially? But I actually just used Twitter's new little feature where there's like you can upload an audio file. Now. Nice. So last night I posted. Oh, my Z shell. Oh, my Z shell. So there it is. That's the official the official word as spoken on Twitter these these days. Yeah. So so obviously an incredibly popular open source project that provides. Well, can you give the elevator pitch for that? And then we'll then we'll get into the topics. Sure. Sure. So much that you're doing, Robbie. The only Z shell is a it's an open source framework for managing your configuration for your terminal. If you're using Z shell as your primary shell. So, you know, these days, like the Mac OS comes with Z shell as a default shell. But prior to that, it was back. And so if you're familiar with bash, as you shellfish, it's a it's a framework for, you know, it's got a couple hundred different plugins and themes that come bundled in it. So that helps developers that are working on specific programming languages or frameworks kind of make some nice little features when it comes to auto completing and just some things that helps kind of add some more delight to your terminal experience and a lot of fun little emojis and colorizing of your terminal prompt. So a lot of customization there. And so it was a project I started a little over 11 years ago, primarily for the people on my team so that I can kind of indoctrinate them to all use Z shell as well. And so and now a lot of people on the Internet use it. Yeah. If you've ever watched a screencast and you thought, wow, how did they get their terminal to look like that? There's a pretty good chance that it's on my Z shell behind the scenes there. Cool. So, Robbie, I'm really excited to talk to you about a couple of things that I think you have a unique perspective. On a unique vantage point. And I want to start with this idea of being a guest or coming into an established code base, because as you already said, you know, that is essentially what you do all the time. You're coming into an established code base most of the time and taking it over as a team, but also as individuals. I'm curious, you know, what are some of the things that you've experienced or that you've seen? Go wrong when you come into a code base as an outsider? Yeah. And I think just for that, that context of, you know, I think there's the scenario we're coming in as a consultant and there's kind of there's let's make an assumption that you're going to be there for a short period of time. You're not going to be there indefinitely. It's not not going to be the same as if you're becoming a full time employee. And I think there's a lot of other things that go along with that. I think we can probably dig into this. Probably definitely some overlap. But going. When things go wrong, usually as a consultant will get pulled in. There's usually a problem, whether that be a resourcing constraint problem on the client side or if you're let's assume Jonathan, you have a company, you're part of a team that has existing application and you're calling company like mine and to come in and provide a couple of developers to help us. So we might be coming in to provide some staff augmentation or to help you fix some maybe very particular pain points within your your your application or your tech stack or. To have you. And so usually if we don't have a very clear goal that is shared with the stakeholders and the developers of the existing internal team, that can be a huge challenge and red flag to work through, because sometimes we'll get hired by the stakeholders because there might be a perception of they've lost confidence in their development team. Whether that is true or not, you know, is it can be hard to determine, but the developers might think that that might be the case internally. They might feel like, well. I don't know who these people coming in. You know, there's kind of this if you think about and think of the movie like office space where consultants come in and people don't really know what they're doing over there and they're coming in like, oh, they're going to get rid of all of us are going to throw everything away or just point out all the problems that we're, you know, that we're kind of that we have here. And so first, I think you need to kind of figure out how to navigate that whole messy world of like, who's bringing me in? Why are they bringing me in? What's their goal? And then make sure you get some time to speak with the other people and other team members to make sure. That you can kind of connect with their goals are as well to the stakeholders goal or vice versa. So things that have gone wrong is where, you know, I think sometimes the consultants will come in and be like, okay, great. I'm here to prove my value as quickly as possible. And so one challenge, I think, to acclimate yourself, not just to the technical and the code stack, but as to the team is to be too quiet. So I think if you're if you're the type of person, you're just like, I can put my headphones on. Give me access to the repository. And I'll let you know when I need some help. Just start assigning me a few tickets in JIRA or whatever tool you're using. And I'll be in touch if I have a problem. And so I think that sometimes can be a problem where because let's also assume that if you're coming in as a consultant, you have maybe a little bit more expert expertise than, say, a junior or maybe even a mid developer, mid level developer. You know, usually consultants are coming because they have, you know, perceived to have, I'm air quoting perceived. Some level of expertise. There is you're coming in to provide some guidance or help solve a problem or you have some experience with some tool sets that maybe the internal team is struggling with. So if you come in as a consultant and are quiet for too long and you you don't really start to foster a collaborative working relationship with the other teammates, there's going to be this mystery about what you're doing. And so the longer that happens, the more people will need to fill in their own assumptions of what you're focusing and working on. And knowing that you're probably being perceived as an expensive person or individual or just a cost to the to the project. Now, there's going to be those sorts of things will start chipping away over time and erode their trust in you. So I think you need to figure out. So I think there's some some strategies to help navigate that. But being quiet for too long is not an effective, I think, way to join a project as a as a consultant and definitely not if you're, say, a new hire. Yeah, that's it's such an interesting problem to solve, because I think, you know, coming in, you have if you are like most people, you don't want to over assert your your newness. Right. And and because it's very easy to quickly kind of ruin the the relationships with the people around you. If you come in and you act like you've got all the. You know. The solutions to every problem that you're finding and the people who you're talking to, you know, just kind of bolster that position. You know, the people that you're talking to have spent more time with this code than you have. So it's totally reasonable to think that, you know, the best thing you can do is sit back and wait. But it's also totally reasonable, I think, and you can speak to this a little bit more to take advantage of the fresh perspective that you have. Right. Your eyes are not kind of used to what you're seeing in the code. And so you have an opportunity to say, hey, has this been thought through a second time? Has this been revisited? And I think that's a huge advantage that gets lost over time. What do you think about that? No, I think that's a really good point. And so, you know, I think there's a couple of things that I I've always been trying to encourage people on the team, my team here when they're entering entering these situations. Is to think about the, you know, trying to think of a good way to explain this. But we have to realize that, you know, people on the client side. So, you know, in our perspective, the client being those with their own product or application that they're bringing us in, you know, those developers. And if you think about your own projects that you've been part of, Jonathan, I'm sure there's areas of the code base in your documentation that you're potentially embarrassed about. Like I haven't had a lot of time to really spend. Focusing on those specific areas of the code base. And I know I need to improve the documentation or there's not enough test coverage there. And you have probably your reasons for why, you know, from the day to day things that pop up, the reasons you have a number of reasons why you haven't been able to get to that yet. And so so there's this kind of like this interesting thing where if like anybody, you know, think about any project you have and like someone's getting hired to come in and help you work on that project. And they might start asking, like, what's going on over here? And you're like, it hits like a little. A weird. Yeah. Weird little sorb. You know, it's a sore spot for you. And you're like, ah. And so you have to kind of internally work through this little hurdle of being like, OK, I need this person is going to help. I need to be helpful. I need to be available. I need to make time for this person. Not to mention, you might have felt like you haven't had enough time to work on this project as it is. Now there's another resource or multiple people potentially joining the project. You need to find time to answer questions that you've already had to answer in the past for other people on your team or what have you. And you're like, OK, how am I going to get my stuff done? And this. And you help this new person be successful as well. And knowing that it's part of your responsibility is a bit of a challenge. So I think as a consultant, you need to know that going in like that person is going to be nervous and about because there's a lot of unknowns. You don't they don't know how you're going to perform. They don't know the questions are asked. You know, they don't know whether or not you're going to write up a report and just outline all the problems in the code and send it to their boss and say, what is going on with this person? This person's falling short. Can hit like those of those. Very. Sort subjects for you. You might be feeling imposter syndrome already, and this person is going to come in and potentially just highlight all those things and you'll be caught. You know, put the spotlight on you and oh, no, like now is your job in jeopardy. So there's there's a lot of mixed emotions going around nervousness. So as a consultant, you need to be aware of that and just come into that space and acknowledge it and try to think, how can you start establishing a some level of healthy trust between you and the other people that you're working with? You might have to do with multiple people. Maybe there's one or two primary people in the project. And hopefully it's more of that than the former, at least in this first several days. So I think some some of the ways that I've I've kind of helped kind of kind of across that that bridge at times is to think about, let's say you're going to come in and you need to get the application up and running on your own development work environment. So say you have a laptop, you come in or they give you one and you're going to go through some documentation and get things set up. So. As you're doing that, you might hit a couple of snags on the way. And obviously anyone will tell you, like, please help us update the README or the documentation if there's some gaps. That's one of the benefits of something new coming to any project, whether you're a new employee or a contractor, is that you can help make those kind of early contributions. And that could be your first pull request or two is just improving the documentation to clear up some things that maybe you hit some bumps. So and then if someone helped you solve that problem, obviously share some appreciation of that person. But another thing that I think often gets overlooked is to if the documentation there you're relying on was really helpful, look at the revision history and look and see who contributed to that and reach out to those people and say, hey, this is some of the best documentation I've seen in a long time. I work on a lot of projects. I just wanted to say thank you and give them some appreciation because then it helps reinforce that not only is the documentation valuable, but they feel like there's now like established and do that with like if it actually is helpful, don't just do it because you feel like it's not. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. that way you can start fostering to get into some of the messier sections later on like we're going to ask like hey i noticed over in this area seems to be lacking some documentation um who would be the best person to talk to you about that um you know is there some any documentation on where this was last when we made when the decisions were made to approach it this way thinking about just being empathetic not only to that person where they're at right now but um to whoever worked on that project in the past because it's easy to go in and just be like this is wrong this is wrong this is wrong and i think we all we all know as developers there's not always a this bullyingness of uh is that a word bullyingness anyways um but a sense of like is something right or wrong is it does it work or not i think often is the more appropriate way to think about that but like is it working anymore and if not what do we need how do we need to adapt and like move this thing forward to better adapt to the realities of the the business today then you know three years ago when this was first implemented so i always think about just that early relationship building to just think about that person is nervous they might feel threatened about their reputation and their livelihood might be feeling at stake because because maybe there's this fear maybe they had bad experiences with previous consultants that came in and just made a mess of things or didn't produce a lot or just came in and just pointed out a bunch of problems and said you know let me know when you want to hire me to fix it so i think you need to go in there with like a different sort of approach to it and i think that's a good way to do it i think it's approach and i think you can apply that as a new hire as well in a lot of those ways i agree with this and and especially you know you mentioned something that i think is really critical to understand as a new hire um and and it also is is critical to another role that we're going to talk about in a minute uh the intern but uh we'll get to that in just a minute the the idea that documentation is almost never a bad idea um and encouraging especially when you find that good documentation encouraging it even further uh this this is something that um of course like you said you know the very few times very few jobs that i've had very few teams that i've joined even for uh you know momentary uh time uh or you know for for a limited amount of time very few teams would say we don't want you to do this we don't want you to do this we don't want you to do this we don't want you to do this to write uh to fill in the gaps in our documentation which tells me two things number one it's something that everybody wants and number two it's a problem that everybody has right it's this is this is a pretty consistent issue that it's much easier to write code and move on to the next problem than it is to reflect and take the time uh once you've kind of completed the task right to to go back and say okay and perhaps it's part of the reason why we have you know certain kind of paradigms of programming like documentation driven design or whatever you want to call that where you have to kind of write out what you want initially you write out the design and document it first before you implement it and then post implementation go back and correct wherever you were wrong when you were kind of predicting that future so it is a it is a problem that everybody has uh and i do think what you're saying about you know coming into a new team and being a part of the design and the implementation of the design is a problem that everybody has i don't even know if assertive is the correct word but understanding that everyone else is you know it's a very content it can be a very contentious moment especially for um for the situation where you have a contractor coming in it may be a little different for someone uh for a new team member but kind of the general principle here is change is scary so whatever team you are joining you are changing that team. You're bringing something scary, regardless of how good of a developer you are, regardless of how much promise you have. No matter what you do, change can be a scary thing for the people that you are kind of, I guess, bringing that change to, right? So even if you are considered one of the best developers, you know, you're bringing in all of this freed up time for them, it can still be something that they're concerned about. And I love the way that you articulate it, you know, to be aware of that, that perhaps even their livelihood is on the line, partially because you're there, right? Would you agree with that kind of perspective that, you know, the teams that you're coming on to, or the teams that you're joining, they may feel threatened by you, even if you are going to give them a better quality of life, better quality of life. Software, whatever it is. Definitely. There's a, there's this interesting balance there of where, you know, you, you've mentioned, like, you want to come in and be assertive, but you can also be over assertive, very, there's like a fine line between this person is just cocky, or I mean, trying to get some better language there, but it just in terms of, you can be a really disruptive person really quickly. And because in a lot of ways, as a consultant, you're given a little bit of leeway to speak your truth, I think, in some ways, because that's what the, you know, the business owners probably want to hear the hard stuff. And you're, they're hiring you to come in and tell them what they might suspect might be true, or, or, or hopefully alleviate their concerns. So I think back to there was a there was a project in the last couple years that we came in. And it was interesting, because there was a, it was a software project where they had someone that had worked at the company. And they had a software project that they had worked at the company. And they had a company for many decades, like, like, I think it was like three plus decades. And so they have been there for a really long time and lots of different technical stacks. And they were they're nearing the end of their, their, their career, and you're going to be retiring soon. And they have new people that have been really, really struggling to come in as employees, and work amongst that environment where there's the kind of the old guard, or the senior old guard. So like, you know, I think we often don't get that scenario where that there's that big of a difference. And just how the tenure of people within an organization, but that, you know, that happens. And so that person's kind of looking at a certain way of like, knowing that they need to pass the baton to the other people on the team. But they don't necessarily agree with some of the new styles and new approaches and, or maybe don't even see the value in some of the, the development methodologies that are being more popular in the last decade or two. And so there's this interesting culture clash, we were witnessing. And so we got pulled in, because we were perceived to have more experience than their internal newer recruits. So maybe we can come in and help kind of create a bridge, like between the person that's leaving, because they know, they need to make sure that this thing can keep operating beyond after they retire. And the newer people can start to start incorporating and take some ownership of everything, because there was a lack of trust of like, well, I'll take care of that, because I don't think they're ready to take take over that feature. Or move that architecture, but maybe, but they felt confident enough to bring in us because they knew who I was back in the day in the Ruby on Rails community, or what have you. And so it was a really interesting challenge where we're came in coming in and okay, process was like a big thing we're trying to help them with. And so trying to help kind of bridge that gap. But it was still, it was interesting that, you know, we had a scenario where we came in. And so we did like, there's documentation updates you can do. But we started making some updates to just like, cleaning up because because we're using some winters and just cleaning up some really basic syntax things, cleaning up some comments in the code. And so we sent a pull request, like within the first few days of just a few little things that someone was experienced, like just some tidying up. And that person had been there for a long time said, Why are you doing that? It's not important. And we're like, like, you need to be focused on these other things. And we're like, but we're here just tidying up while we're going. Why this took like 15 minutes? Why is why are we having a 15 minute conversation about such a small thing. And, and we're realizing we're still having to like, we're, we immediately fell into the same sort of scenario that the other people that had been hired full time there were starting to experience like, so now we're like, okay, we're now we're part of this weird situation of trying to figure out how we're going to even just change the culture of this team. And when that so that that type of scenario could be really interesting, too. But it's, I'm trying to get back to what like my original thinking for telling. that specific story it was, but I don't think it was so much that they thought that they were worried about their, their livelihood, it was more about their legacy. And, and how much they had helped that organization for multiple decades get to where they were. And then to feel like in some ways, like, the changing of the guard was going to kind of remove them in some way from and their, their impact or their value. Yeah, yeah. And so that was a, that was a tricky one. And, and, and immediately, we didn't, we weren't super successful. We worked with them for a number of months, but it wasn't, it wasn't something that we were really able to push through and get to the other side with. And so, and then COVID happened. And so I think there's some challenges related to that as well. But the, but I think about those types of things and be like, oh, that, how do we, if anything, I felt a lot of more empathy towards that individual and thinking, okay, how am I going to bond and establish a trusted relationship? So we can do these sorts of things without them being kind of doing the same thing that they would do with their junior or mid-level developers being like, don't focus on those things. But we're like, but those are the things that keep contributing to a lot of the problems you're having, you know, six or 12 months down the road. And so you have a mess, we need to help clean it up, but you don't want us to focus on the mess. You want us to build a new feature because that's what the stakeholders want. But we, there's kind of like a tricky balancing act there to kind of navigate. Yeah. With Linode, you can simplify your infrastructure and cut your cloud bills in half. With their Linux virtual machines, develop, deploy, and scale your modern applications faster and easier. Whether you're developing a personal project or managing larger workloads, you deserve simple, affordable, and accessible cloud computing solutions. You can get started on Linode today with a hundred dollars worth of free credit. That's a hundred dollars in free credit. And you can find all the details at linode.com. Linode has data centers around the world with the same simple and consistent pricing, regardless of your location. You can choose the data center that's closest to you. And of course you receive 24 seven top tier support. Here's the thing. Linode doesn't have any tiers of support, no handoffs, regardless of your plan size. So you can choose shared and dedicated compute instances, or you can use your a hundred dollars of credit on S3 compatible object storage, managed Kubernetes, and more. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. If it runs on Linux, it can run on Linode. Head over to linode.com slash developer T and click on the create free account button to get started today. Thanks again to Linode for sponsoring today's episode of developer T. This actually speaks to another question that I have about, uh, um, the consulting vantage point that you have, because you've had a chance to see a lot of these different junctures for businesses that are critical enough that they've decided to make, you know, this kind of, you know, you know, you know, you know, you know, you know, you know, you know, you know, you know, I wouldn't call it, not necessarily a drastic decision, but certainly a change, a juncture in that business's life cycle that they've decided to make a change. And I'm curious about a couple of things about that juncture. The first one is, what is the most common reason that you see people needing to bring in outside help, whether that's augmentation, like you mentioned before, or something is going wrong and they can't put their finger on it? What are those types of things that you start seeing consultants like your company being brought in? What is the most common couple of things that you see? Yeah. I think staff augmentation is the most common that companies come to us for. It is the thing that we do least often, usually just because we're more of the, that usually involves like a longer term, like 100% someone on their project for six, 12 months at a time. And that's not quite what we're- I don't mean to interrupt you here. I just want to kind of clarify for people who are listening who may not be familiar with this, the idea of staff augmentation and kind of what does that provide to a business? Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. can you kind of just give the you know the very acme version of what that is well in a in a in a really high level it's kind of almost like you're acting as a very specialized staffing company where a company a large company or even small company might be looking at their roadmap like we want to build up a lot of features over the next six months we don't know if we're going to need people beyond that six months at this point and we might want to hire people of a certain skill level to help us get there maybe quicker and so they'll look to consultant companies like us to say hey can we get a two senior developers that have at least x number of years with and have worked with these specific technologies to work on this project for the next you know full time for the next three six months or what have you and so then they kind of just get embedded in the team and work on that project and then once that contract is up you know then they get you know we put them on other projects and so the kind of benefit to the company there is to it's not a it's not a permanent decision the other part is if it's not working out within a certain period of time you could probably pull the plug and you know not be on the hook for having to worry about everything else when it comes to you know all the stuff that the company needs you know like they would normally need to think about with a full-time hire when it comes to like benefits programs and everything else in terms of you know like what they're you know even just like having to provide you know like one-on-ones maybe it might not even be a full-time hire but even be need to had or uh i'm just thinking about all the the ways you would need to get rid of a developer if they weren't like a good fit for your company and need to terminate that relationship it's it's different so it's kind of like we kind of take that risk on as our company and so we're the ones that kind of do that but to have to navigate all that so it's i think at a really high level that that's the the basic thing is to help move yourself faster and not be making a permanent decision so you can kind of bring in that expert talent that you might not be able to get by just doing it on your own but you can kind of bring in that expert talent that you might not be able to get by just putting up a job ad and and finding that right like who are so i think it's a way to speed up the um you know maybe overcome some of the higher the the how long it might take to hire and bring people on so like this happens every year like they're all usually there's a couple um organizations that we know that their busiest time of the year is like because they're in e-commerce so like between in the u.s between thanksgiving and you know christmas is like their big season and so they'll usually contact us september and so they'll usually contact us september and so they'll usually contact us september maybe october saying hey can we get a few people for that period of time we or from now until the end of christmas so that we have some extra people that have a lot of experience that we don't have to spend a lot of time you know ramping up um and we don't we don't know if we're going to need them you know in january and so it's that's it's a way to kind of like add some velocity to your project and then kind of quickly pull that off and not um still be responsible for keeping figuring what those people are going to work on after that yeah i think that's a good point i think that's a good point yeah i've heard it described as kind of a faucet where you can turn on the resources or turn them back off um without you know having to have the overhead of finding individual contractors and um you know especially if you have a pool of resources so let's say you you know maybe you start with one developer and you evaluate you know the quality of the work and to make sure that things are going well and then you can scale up to two or three all within the same company you know that that is kind of the idea that at least the way that i understand it so that you can uh instead of having to you know negotiate each individual contractor's rates for example or evaluate each individual contractor's quality of work you have a little bit of a buffer by working through that staffing yeah or i guess uh staff augmentation layer and those people but you mentioned that that's not really the case right now i think that's the thing that's really what you guys are doing most often uh what what are you doing more often than that well the the thing that we end up typically you know outside of the taking over projects and being the primary development team of those projects which is like kind of the core of what we do um and those are usually for like projects that are like there might be a large company that they definitely have their own internal development teams but this project is maybe too small of a thing for them to have a team for so rather than having like one person to do the work that they want to do and working on it they'll kind of outsource that to a team i guess to be responsible for it but a lot of the reasons other companies will come contact us as well is because they're behind on the versions of their application so they need help with uh either with helping perform and upgrade on say the foundation life their application so we primarily work with urban rail so there's a lot of like upgrade related projects and something that we've been doing more of and focusing on is not only maybe just doing the projects because actually we've started to take this opinion that we don't think we should be the team that handles like shouldn't outsource entirely an upgrade. I think you might probably would probably understand why. But there are people out there like, well, just like we don't have time to do it right now. So why don't we hire a team that's really competent at that, let them do it, and then we'll deal with the big merge and, and then we'll be done with it until it needs to happen again. And so one of the things we've been trying to focus on the last year is working with teams on helping them build up those skill sets themselves so that they're regularly thinking about how they're going to keep their dependencies and underlying frameworks up to date so that they're not falling for five years behind and then needing to call a company like us when they're when they realize that their underlying version of Ruby on Rails or Ruby itself are no longer getting security patches. And that raises some alarms on their end. So we just quite often, it's usually like, oh, no security problems, because we're not getting security patches, we need to call companies like Planet Argonne to come help us and help get us out of that mess. But what we want to do really is like help them fix that problem. So it's a they can keep it going themselves long term. Yeah, yeah. And so so it seems like it's these, these two to the, to the name maintainable, it's about maintainability. And, you know, that brings me to perhaps the kind of the closing question on this particular, you know, in this section, I guess, of the interview. The, the idea that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, that, software is unmaintainable. In what ways do you see most commonly software becoming poorly maintained? And I guess, kind of what I'm, what I'm hoping to hear from this is not necessarily a specific thing, but, you know, a kind of category of things, what are people doing? What are, what are those, I guess, the behaviors, the habits that engineers may have, you know, may have, may have, may have, may have, may have, may have, may have, may have, may have, may have, may have, may have, may have, may have, may have, may have, may have, are ultimately practicing in that leave their software unmaintained? What are you seeing most commonly with that? I'm glad you asked that. I think they're, you know, I talk to a lot of people on the podcast and get their interpretation of what they believe is maintainable software, but thinking about the contributing factors to it. And in my own experience, there's a tendency for developers to think I think to not really stay consistent to some degree with their habits. And I think when it comes to like, okay, I'm going to document this or I'm going to like, you know, I think especially on projects early on in their, you know, in their life cycle where you might be building an MVP for a project and you don't even know if it's going to be around any year or two. So there's always that balance of like, you're trying to build something up and you're going to see if it's got a product market fit. And then maybe the team will grow in the future and, you know, get some investment or you actually make money with the product and you can build out your team and expand it. And that's a great situation. And then, so then you have to come back and then usually start backfilling and figuring out how you're going to clean up things or help things scale. And so, but sometimes there's a lot of projects out there that they don't really end up scaling a lot in terms of developer team size. And so they're kind of, there might be a really small team. And so whoever those original people on it, they kind of set the tone on projects for how, how much they're going to invest in things like testing, documentation, just consistency in the code, or even just like how, you know, if you think about even like the simple thing, something that I always look for, like even just like when you look at someone's, like someone applies for a job and you look at their portfolio projects and you go look at their own, like say GitHub profile and they'll see the projects that they worked on in their portfolio. Just go look at their Git commit messages to get a sense of how are, how do they, how do they operate and deal with, how do they document how, why they're making changes in their Git commit messages themselves when they're kind of working on their own to get a sense of like, you know, I think, I think that can tell a story. It doesn't mean that there, there's anything necessarily inherently bad with that, but I think those types of things happen on real world projects a lot too. And, you know, there's a, I don't know how many times you've, you've might've seen a project where you look at the first, you know, 50, 100 comments or Git commit messages or whatever versioning, you know, you're using and there's just a lot of like, really one, two word type thing, fixed typo, you know, updated whatever. And there's not any explanation of what, why those things are changing. And so I think you can set a tone with, as you bring other people into a project where the, the patterns that you've established is what people kind of learn, or keep building off of, unless you get hired, unless someone comes in that has more experience and says, okay, this is, we need to improve upon this. And they make a constant, decision to change the nature and the, let's say the little microcosm of the culture of that specific application. And, you know, if they start making some intentional choices, like we're going to start doing this now, we're all agreeing to do this, great. And you start moving in that direction, but you will look at code where, you know, there's always, you know, there's conversations about finding the, let's say the God objects in your application, where there's just too much complexity. And like there's all high churn in those areas, there's a lot of really big objects in your application. And so when you bring in other people in the project, and they're like, well, this seemed to be where people added things last time, I'm going to add something else here. Then it kind of just keeps reinforcing that that becomes a bigger mess. And so, I mean, a lot of coding is done to some, to some degree can be copy pasting, or at least like copy and pasting an idea in a very similar area. And it might not be the best long-term approach, but we, we take those kinds of easy, easy outs a lot in that all those little micro decisions start to add up where five years down the road, you're like, wow, there's so much, this is everything's brittle because there's maybe a lack of test coverage, or there's like just a lot of interconnectedness in the code that, you know, if I don't, people don't really have a good holistic view of it anymore. You know, we talked about documentation can also get really outdated. You know, there's like schema diagrams that people might've produced and nobody keeps updated. They might not even be valuable. I don't think people sometimes consider deleting documentation enough because it feels like, like that might be not helpful or they might be needed one day, but sometimes all that extra noise can also be distracting and make things more difficult. So people become less reliant on it because there's not like a, like if people start to dismiss documentation as, well, that's, that's outdated. I'm like, why is it existing anymore? This let's just get rid of it then in the same way that we should feel comfortable deleting code at times, if it's not being updated, it's not being helpful. If it's not being used, let's get rid of documentation and other resources that are just noise. Because I think the noisier things can be in an application environment. The harder it is for new people coming into the project can really go, okay, this is how this team works and operates. But if you, there's an undercurrent of, well, the culture in this code base is to just leave stuff. Okay. Then I guess I'll keep doing that. And that's an easy decision to make. And you don't have to, people don't have to think too critically about it. So I think those are some things that I see that are minor, but they do kind of stack up over a period of time. Yeah, yeah. You know, I think it's really interesting that you mentioned going and looking at GitHub commits. There's this difficulty of trying to understand what is going on on a team. And this is kind of the underlying theme of what we're talking about in this whole episode so far. If you're joining a team, whether you're joining as a contractor, if you're joining the team long term, if you're coming in as a manager even, or let's say that you're getting moved from one team in an organization to another team in an organization, people who have experienced this, they understand for sure that trying to get an idea of what is actually happening on that team, it seems like it would be the easiest task in your onboarding checklist, right? But this is the one that takes the most investment. Trying to get a handle on what is the history of this team? What do the people on this team care about? What do they not care about? What are the kind of consistent problems? What are the dead angles, the missing information? What do they not know about themselves even? And what are they kind of misunderstanding about the society? Yeah, yeah. I think that's a really interesting question. I think that's a really interesting question. I think that's a really interesting question. I think that's a really interesting that they're working on. As a new person coming in, it's easy to have misconceptions, very easy to have misconceptions about the people on the team already. For example, I've joined teams where I thought, wow, the engineers on this team know this code front to back. Every line of code that's here was put here intentionally. And me walking in today, it has, you know, I need to, like we said earlier, I need to take a step back and just listen. That's almost never true, right? It's almost never true. And so it's, and again, especially if you're a junior engineer coming onto a team, it's very difficult to know what's actually happening on that team. It seems like it would be very simple, right? You know, and your point of looking at, you know, get commits, this is a really good example of go and try to find data about what's going on on this team at the individual atomic level, right? At the smallest level possible, what kinds of changes are being made to the code? Is there improvement type changes or are there quick fixes? Is there a lot of kind of responding to errors? Is it, you know, what is the common pattern? You can kind of think about this as like the direction of the wind in a way, right? Like that's which way is the wind blowing? Oh, it's blowing in the error direction, right? Like we can kind of think about that. You know, the code that's going out as responsive to errors right now, or you can think about it as, well, the wind is actually going down the maintainable track. We're adding to the test base. We're, you know, improving, we're doing refactoring. We're all in that improvement space. Or you can think about it going in the direction of new features where that's being driven. And it's, it is interesting because even in the most hyper optimized teams that have everything in place, you can think about it as, well, everything dialed in, they've got, you know, the team is gelled. There's the, the, the, you know, everybody's happy with the team. They're happy with their production. There still is a balance to strike between, you know, refactoring, maintainability, et cetera, and new stuff, right? New, new features, new code. So I'm wondering how do you, especially with your kind of bias towards maintainability, how do you see that playing out on a healthy team? The balance between those two things, newness, new features, all of that stuff, and the necessary and critical maintainability that acts as a basis for that new work. It's interesting in that. I think I have a, a skewed worldview on that. And cause I don't, I don't work on a product team. Um, a lot of the projects we work on where we're consultants will be, they might be a product type of organization or have like a SAS, but a number of our projects we work on, um, it's not a lot of really there's, there's some element of the, the, the drive from the, our clients is not necessarily always new client or new features as much as it is maybe improving performance or just keeping things updated and maintain. And so there's periodically we new features and such that are so quite often, you know, depending on the project we're working and we're responsible for, there could be like, Oh, we're doing a really good job. We're doing a really good job. We're doing a really good job. We're doing a really good job. Because we're, we took over a project that was kind of buggy. It's less buggy, but a lot of the work we're doing right now is bug fixes or data, um, tracking down data problems and helping like improve the stability of their data and in their system and helping make, make more sense of that. So it's an interesting one. I don't feel like I have a good grasp on how to like come up with like a healthy ratio there, given that I don't work in, I think where a lot of companies might be, or a lot of people listen to me. I don't feel like I have a good grasp on how to like keep things stable and, you know, iteratively improving upon things. And so when we get to work on features that can be great, but we don't always get to focus on that necessarily. So, um, because stability and, and taking care of a lot of like the support stuff. So back in there, we're providing maintenance and, you know, occasionally for those, you know, you're going to have to take a lot of time and a lot of time and a lot of time projects, we might have a handful of features or build outs that we're doing for those projects to extend what they're capable of. But that's not always the primary focus of why our clients are hiring us. Yeah, that makes total sense. And, you know, to, you know, having been on many product teams, I know that this balance is really hard to strike. And part of the reason for this is purely because of the way that decisions are being made about products, right? Most of the time, somebody like an engineering manager and somebody like a product manager or product owner will meet and determine priorities together. This is a very common pattern for while the engineers are working on, you know, the current sprints, or whatever your iteration is called, your kind of grouping of work, the engineering leader, and the product leader are meeting together to talk about what's coming next. And the engineering leader, their job is to try to understand and cut through all the haze about which of these maintainability issues is going to cripple us in the long run or in the short run. And how do we, you know, prioritize that versus these new features, right? And this is notoriously a huge problem in startup world. But it also pervades in non-startup companies, because the perception is very easy to say, I don't see anything wrong, right? It's very hard to see those invisible issues that engineers are very aware of, but products, perhaps product owners and the product organization or sub-organization, department, whatever, they're not necessarily seeing the pain from those. So finding ways of elevating, what those maintainability issues, what are they going to cause in the long run, right? In a very simple way, you could say, okay, well, we have security problems that are leaving us vulnerable today, right? That's a very simple one. What are some other ones that you've encountered? Yeah, so there's the security is always like one of the best ways to kind of at least help, I think, in some ways, like help pitch or sell, dealing with like, upgrade to type related projects in general. But there's also, you know, this could be, we can improve the performance of the application so that, you know, the application will run a bit faster. And that's going to be, you know, that's proven to be more effective on mobile devices when people aren't on a Wi-Fi connection, or, you know, the, I think another part of it, if you can kind of, there's always a business value associated to the work that you're doing. So if you're going to make some improvements or to refactor something, it could just speed up, you know, the development life cycle in the long run, or like, or if you can calculate, we're having to work around these problems. And then every time we have to deal with this area of the code base, you know, it's costing us a couple of extra hours or a couple or days to navigate that or because it's because we lack enough testing in this area of the code base, or it's just inefficient with how we're trying to navigate things. And we have to do some like data migrations to make to anytime we make a change in this area. And that needs to be tested. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. If we can refactor this area, we think we can remove some of that decent percentage of that extra work that we feel like we're doing right now. So that can be another way of like helping, like we're removing obstacles. We're going to save time in the long run if we invest time now to tackle those types of challenges. And another thing that often gets overlooked, I think can be really effective with the right stakeholders would be to talk about employee retention, that if you're dealing with an older code base, you're going to have to take a lot of time to take a lot of time to take a lot of based or a very painful to develop an environment where you feel like your team has high churn in terms of needing to bring on new people. So it could be the ramp up time for onboarding is super slow because of the way things are. Like we need to improve some areas of application because it'll help speed up how quickly we can introduce new people into the project. And or we're losing people because they're excited about the opportunity to get to work with their new versions of some of these technologies or other technologies potentially entirely is often a reason why people go move out elsewhere. And so like if you stay to like rest and too much like, well, we don't want to invest in improving things, you might lose your people and all the knowledge that they currently have of your infrastructure and application will walk out the door with them. And, you know, you might have a good process for like document everything before you leave. But we all know that not everything. It's documented because we don't know what everything that's in each other's head that we need to gather from them. So that I think that's oftentimes a thing to think about is like we if we fall too far behind on some of these problems or if we don't improve these processes, we're going to lose people. And that's going to cost us in a lot of other ways, too. I totally agree. Totally agree. And there's this, like I said, is a continuous problem in in product. And I think that's something that we need to be thinking about. And I think that's something that we need to be thinking about. And I think that's something that we need to be thinking about. And it's necessary for engineering leaders, especially to learn what are the most important kind of shared concerns that I can focus on to make sure that our software doesn't fall into a very difficult state. One of those is certainly hiring, right? Or we're not necessarily hiring, but although hiring is another one, right? But, you know, if you were out recruiting and you said, hey, we have a policy, for example, that we focus a certain amount of our time on maintainability issues and upkeep and sweeping out from under the rug, so to speak. That is a very attractive prospect to most engineers. I think it is. And there's also this idea that there's another expectation that I think sometimes where companies could also. In some ways, I've seen this where I think developers can also and I know that the audience is primary developers, but they can allow their own kind of personal interest of learning new technologies, help make technical decisions that might actually be really harmful to the organization that they're working with, too. Like, oh, we're going to we're really interested in using microservices, but you're a team of three and you spend a bunch of time building out these things and start going down this path. And then they leave and then you or maybe they don't leave yet, but we'll see. We get called in to help out companies where it might be a small team, but they have a lot of things that they're really interested in exploring. But some of those people aren't there anymore. And it's only like a year after they made that decision. And they're like, we don't really know what's going on. The developers said they wanted to use this new framework. And so they built up. I'm like, well, this feels really complicated for what you're really doing. And it starts to feel a little bit like resume driven development. I kind of took over for a bit because the stakeholders would trust the developers to make good decisions. But I don't always think that developers are always acting in the best interest of the company as much as they're also looking to scratch their own itch. And that could be a really challenging thing to navigate. And so the people that have to come in and help navigate taking over those projects are in kind of for a fun ride of having to kind of make sense of all that, too. And so which might be a way to recruit new people for new technology. But then, you know, if you're also learning. The new technology on the job and you don't have anyone else helping establish good good foundations for what your workflow is going to look like, your development patterns are going to be or making that the elusive another back to the whole theme of maintainable is one of my goals is to fight the big rewrite because oftentimes rewrites fail because there are people who drastically underestimate what it really takes to go through to not only build out a new application, but to keep the existing one going in parallel and then dealing with that migration. At some point. And so, yeah, a lot of, you know, a lot of failed upgrade projects out there. A huge thank you to Robbie for joining me on today's episode. If you don't want to miss out on the second part of our discussion, go ahead and subscribe and whatever podcasting app you use to listen to developer T. The next episode of the show comes out on Wednesday. So if you don't want to miss out, make sure you subscribe this episode and every other episode of developer T can only happen. Because you listen and you share this podcast with other people. You sharing this with someone that you believe would appreciate it. This episode specifically this episode, that's going to help this podcast grow because you may find someone who decides to also continue being a listener of the show. And our listeners are the reason we do the show. If we don't have you, then there's no reason to do the show. Seems pretty simple, but no pressure. The fate. Of the show is in your hands. We have no plans of stopping anytime soon. So thank you so much for listening. Thank you for leaving reviews is another way to help us out. Of course, this episode and every other episode of developer T can be found at spec. Dot. FM. There's show notes there. There's show notes also in your podcast app. Most likely if you use a normal podcast app, the links to our sponsors are there, including today's sponsor. Linode head over to linode.com slash developer T to get a hundred. Dollars. Yeah. Worth of free credit as a listener of this podcast. This episode is produced by Sarah Jackson. My name is Jonathan Cottrell. And until next time, enjoy your tea. .