In today's episode, I interview Sam Lambert, Director of Systems at GitHub. Today's episode is sponsored by Hired.com! If you are looking for a job as a developer or a designer and don't know where to start, head over to http://www.hired.com/developertea now! If you get a job through this special link, you'll receive a $2,000 bonus - that's twice the normal bonus provided by Hired. Thanks again to Hired for sponsoring the show!
In today's episode, I interview Sam Lambert, Director of Systems at GitHub.
Today's episode is sponsored by Hired.com! If you are looking for a job as a developer or a designer and don't know where to start, head over to Hired now! If you get a job through this special link, you'll receive a $2,000 bonus - that's twice the normal bonus provided by Hired. Thanks again to Hired for sponsoring the show!
Please take a moment and subscribe and review the show! Click here to review Developer Tea in iTunes.
Hey, everyone, and welcome to Developer Tea. My name is Jonathan Cottrell. And in today's episode, I interview Sam Lambert. Sam is the Director of Systems at GitHub. And if you haven't used GitHub in the past, you probably will in the future. At the very least, you've probably heard it. You've certainly heard it if you've listened to every episode of Developer Tea. So I'm really excited to talk to Sam about scaling such a large operation. Today's episode is sponsored by Hired. If you are a designer or developer and you're looking for a job, check out Hired.com. Companies like GitHub have actually used Hired to find employees to work in their company. So go and check it out, Hired.com. We'll talk more about what Hired has to offer Developer Tea listeners later on in today's episode. But first, I want to jump straight into this interview, which I did live at GitHub. Thank you very much to GitHub for bringing me into your office and allowing me to check out all the awesome stuff that you have going on there. I really appreciated the time there and I appreciated getting a chance to meet Sam in person and do this on site. So thank you again to GitHub for bringing me into your office. And once again, let's go ahead and jump into the first part of this interview with Sam Lambert. Sam, welcome to the show. Thank you. I'm excited to have you. I'm ready to talk about GitHub systems. Sam, why don't you tell the audience of Developer Tea what your position is at GitHub and what you do for GitHub on a day-to-day basis. So I'm the Director of Systems here at GitHub working with and for the systems team. The system team is primarily responsible for the scaling of Git, scaling our data stores, such as MySQL, and basically doing... general systems engineering around our stack. So the skill levels range from kind of C engineering to then running sort of MySQL and databases at scale. So we'll spend our time doing various different things and helping out in various different areas of the company. But a large part of our daily work is around scaling Git and keeping Git available to all of our users. So we run a large amount of infrastructure to support storing as many Git repositories as we do. And the kind of the scaling and the proxying and the working, you know, basically making Git available across our entire fleet of servers is the responsibility of the systems team. Do you have maybe an estimate that you can give listeners of how many repositories at this moment there are? Is there a good estimate? So we're above 30 million projects on GitHub.com right now, and that number is growing significantly day on day. So, you know, there's... There's a huge amount of repositories, a large amount of users. The use cases for this workload as well are very varied. We, you know, we run anything from CI farms, you know, hundreds and hundreds of hosts connecting to a single repo, downloading the code, running CI against them to individual users using GitHub every day for their just normal workflow. So among those above 30 million repositories, we have different workloads and different huge open source projects to private... small repositories that people are running. That's great. I know I've used GitHub for... Oh, I've been a member for at least eight years now. So since I was in college, actually, I've been using GitHub. I was introduced to it by a company that I was at an intern... I did an internship at... Oh, wow. And this was when I had first learned a little bit about front-end development, and, you know, I was pushing things through FTP and that kind of stuff. Oh, yeah. Say upload on save. Yeah. It's great. I've done such horrible things as well. Sure. Well, and the funny thing is, you know, there are people who are listening to this show, and this shows how large of a field this is and how varied it is. There's people who are listening to the show right now who surprisingly are still using that method because they haven't really been shown the better way. And when I say a better way, I mean, you know, anything that saves versions pretty much. But, you know, for those who are listening, I think it's worthwhile to mention that there is, we may need to disambiguate between Git and GitHub. So, Sam, you can probably actually do that more eloquently than I can. Yeah, absolutely. So Git itself is an independent open source project. GitHub contributes to Git. We have core Git maintainers working for us, and we've pushed a lot of changes into Git. But Git as it stands is an open source project. We host Git. We build on top of Git to allow you to collaborate using our... software and our site. But fundamentally, Git itself is a standalone version control system. And then we use that and GitHub to form the basis of what we do. Well, I guess it is definitely the most popular repository SCM, right? Yeah. The source control. And possibly partially to do with the fact that GitHub exists, right? GitHub pretty much requires that you use Git. I'm... Is that... Is that true? Do you... You guys don't allow for... We do actually, quite interestingly, you can transparently use SVN through on any of the... any repository on GitHub. You can... You can access it via SVN. Okay. You can also import from other version control systems into GitHub. Very interesting. Yeah. But the default, the thing we are promoting is essentially the use of Git. Whether or not you choose to use SVN or... And people do. In fact, an engineering... One of our engineering blog posts came out today on how we build that... how we maintain that SVN bridge so that people who do use SVN can still get the benefits of GitHub. And Git actually wraps SVN, doesn't it? Doesn't... Isn't there... There's not a SVN wrapper in Git? I was under the impression that there was a way to do SVN work directly in Git. But maybe not. I think there could be projects that would allow you to interface, but I'm not sure about what you're referencing necessarily. See, I'm one of the developers who was introduced into source control through the lens of Git. So I don't even... And have experience with these other ones. There's definitely scripts for trans... Like I worked at a company that moved from SVN to Git, and we used... There's numerous amounts of scripting you can to support between the two. But primarily Git is completely standalone. So I want to kind of back up a little bit and ask you about how you got to where you are now. Because, you know, an interest in systems... You didn't just wake up one day and say, well, I guess today I'm going to start building systems. So how did you arise... How did you arrive as, you know, a systems engineer? So I kind of fell into it really, actually. I was doing development years ago at a small web company that was doing like e-commerce sites. And there was no one really maintaining the servers, looking after things, the databases. And I kind of just... I really just want to work on the things that are most impactful for the place that I'm at. What that looks like, it doesn't matter to me, really. As long as I'm making impact, I'm happy. And that was the place I could make the most impact. At Atomic Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp Camp like data analysis, large-scale systems. And that went from something tiny to something huge. And that's where it really changed for me. It's like we had this need and this necessity to understand systems at a deeper level. And that's where the interest really started to grow to me. For me, it was just really, really enjoyable. I specialized in databases. And that's how I came to GitHub. So GitHub didn't have a DBA. They were looking for one and I applied, got the job here, started doing database work here. And then my interest started to wander a little bit into more elements around, you know, how can we make positive change as a group? How can we work on things like making the site more resilient, more available? Started introducing some processes around how to do that and different ways of thinking. And it sort of moved in from that into kind of a leadership role within the company. And so now that's how I'm doing what I'm doing. So telemetry, yeah. Just for anyone who doesn't understand what the term telemetry means, I think that's probably one of those terms that could be confusing. So yeah, I can't recall the dictionary definition, but really it's collecting data on a remote device and sending it back to a central location. So think of like telephone, right? Yeah, there's telemetry everywhere, really. Like, you know, I remember at one point we were talking to people that would do telemetry on like floodgates so that the floodgates could report back the status of what they were doing. Like telemetry can be absolutely anywhere. Cars run a lot of telemetry now and essentially they phone home and gather data and bring it back. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. Yeah. At an industrial scale, I guess. At an industrial scale, I guess. In our case, it was electric vehicles. They were commercial electric vehicles. They were huge. And it was a very much experiment to get these out among drivers and understand how they were being used. And at that point, there was a lot of misconceptions about electric vehicles and how they worked. And we were able to use data to actually to solve some of the problems drivers were seeing. And it was fascinating, actually, to see that type of data centrally. Today's episode is sponsored by Hired. On Hired, software engineers and designers can get five or more interview requests in a given week. And each offer has salary and equity up front. Now, Hired has full-time and contract opportunities. So it's not just for people who are looking for a long-term job, but also those who are looking for the jobs that will fill in the cracks between other contract opportunities. Users can view these interview requests and accept them or reject them. Before you ever talk to any company. So you as the developer, you never have to have any awkward conversations. Hired works with over 3,000 companies from startups to large public companies from 13 major tech hubs in North America and Europe. And perhaps the best part for you is that it's totally free for users. Now, if that didn't already sell you, then perhaps this will. Hired normally provides a $1,000 thank you. bonus just for using their service once you accept a job through Hired. But if you use the special link that you can find in the show notes at spec.fm, then that bonus doubles to $2,000 just for using the special link in the show notes. That special link is Hired.com slash Developer T if you want to go straight to it. But of course, once again, it will be in the show notes at spec.fm. And thank you again to Hired for sponsoring Developer T. Go and check it out. Hired.com. Check it out. Hired.com slash Developer T. But I'd like to know a little bit, and you've already shared with me about the remote working culture just a little bit. But I'd love for you to kind of talk about what a typical day at GitHub looks like for a young developer. Let's say I was to get hired at GitHub tomorrow. What would my job look like? So I think it depends on what area of the company you're working in. And also, there is no real typical day. A lot of the work people do is very much self-guided. We have... The company is largely distributed. Over 50% of the company aren't based in San Francisco. They're based around the world. And people working in different time zones. And just different situations, really. You know, there's members of the... We have certain members of the team that are nearly nocturnal. They work in just... In the hours that make sense to them. When I was... So I started my first two years at the company was from... I spent my time in England working from there. And I would just work... Even I could just... It was great. I would work at times that were sort of... Would fit around my life as well. And, you know, I enjoyed working in the evenings. And so, really, there is no typical answer. And that's what makes GitHub really interesting and fascinating to me. People... They work hard. People dedicate themselves to this company. But it doesn't necessarily represent itself as showing up to a desk between 9 and 5. There's activity on weekends. People might feel a flurry of creativity. And they wake up and they just get going and start doing things and start building. So there's no... Real prescribed way of working. Yes, there's no typical. We have a chat room that... We have chat rooms that we're continually chatting in. That's where... It seems to be... That's where the company happens, really. Even though we have members of the systems team that are based in San Francisco, we tend to do our work asynchronously through chat, through pull requests, through issues. Because if we were to sit in a meeting, it would be about us and not about the people that aren't there. And those people have valuable contributions to make. And we need to make context to provide. And so doing it within chat and in an asynchronous manner benefits all of us. This is almost this contract that... Sort of social contract that... If you send a pull request, you're not expecting... For you to be productive, you're not expecting someone else just to be the other side, ready to just respond to you right away. You have to manage your time and what you do in a way that allows contributions to come from people all over the world who may or may not be working that day, who may or may not work in the same... A similar schedule to you. Mm-hmm. It's just kind of the way we do it. And it's actually... Once you get used to it, it's a fantastic and empowering way to work that's also extremely productive. Yeah. Yeah. You said quite a few things there that we could spend all day talking about. I think there's a lot of approach to working environments that kind of... The idea, for example, you mentioned that you may have a flurry of creativity over the weekend. Um... That's something that I've talked about on the show before, the idea that we have kind of seasonal working habits, right? So you can go through a week where you're not very productive at all, and then you can have five hours of super productivity, and then you go through another week where it's just pretty normal. Mm-hmm. Or you can go through a whole season where you're either only kind of productive or... You know, the idea being that you can't really expect an exact amount of productivity out of somebody, you know, in a given period of time. It fluctuates. People fluctuate. Yeah. And so it's interesting to hear that, especially when you have team members that are remote. You know, how... I guess what I'm interested in hearing from you now is, how do you ensure that, you know, people aren't taking advantage of the system? And I would assume that this is something that you kind of take care of in the hiring process, right? But the flexibility is such an important piece of the remote culture. You know, how do you ensure that... As a culture, everyone kind of agrees that there is a certain level of productivity that we are, you know, shooting for. Like, what is the goal, I guess, is what I'm saying. Yeah. So I don't think we guarantee it to a level above any other company guarantees. I don't think sitting at a desk for eight hours a day is a guarantee of anything. You can game any system. If you're not interested, if you're not engaged, if you don't care, you're not passionate, it's fine. When I've worked nine to five jobs, in the past that I wasn't passionate about, I did my work, I showed up, I had unproductive days, unproductive days, just like everyone else does. I think we do it the same way everyone does. And we have a strong peer feedback culture. We give each other feedback. We understand the standard that we have for contributions. And it's very difficult to quantify anyone's value, absolutely. And the difficulty doesn't get... There's no more difficulty. There's no more difficulty to do that in our culture than I think in any work culture, really. It's about, is that person making positive change towards the goals of the company? Does the person's manager, does the person's peers believe that that's happening? And is it being seen? Then that's all we need to worry about, I think, in that sense. So pay attention, people who work at smaller companies. GitHub is a very large company. And in relation to the average web development firm, for example. And the quote that I kind of pulled out of that is, the nine to five is not a guarantee of anything. The eight hour workday is not a guarantee of anything. It feels like, well, it really works as kind of a security blanket. We create this system, this process where everybody follows it, and that makes it measurable or predictable in some way. But the reality is the output isn't necessarily guaranteed. And really what you're hiring is a person, not a person's time. Yeah, absolutely. You want, once that person is understood as someone that you want to work with, it's then about optimizing for their creativity. And by not dictating a single workflow at GitHub, we optimize for a range of people. I'm not saying this is the best workflow for a legal team. I'm not saying it's the best workflow for a HR team. I'm saying for the engineering side of the company, this works well. And we don't dictate that everyone works the same way. There's just different workflows. Mm-hmm. And as someone who works in leadership, my workflow is highly synchronous. I spend a lot of time talking to people. And I feel I benefit from being located near the HQ. But it's possible without that. There's no use dictating a workflow to anybody. And this seems to be the one I believe is the right workflow for engineering teams. My views on that may change as the company shapes and grows. I don't have a dogmatic approach towards any workflow. I'm happy. And I'm open to it. Yeah. As things evolve. And I've watched the workflow evolve at GitHub over the years. And it's exciting to see how it builds. Like, it's exciting to see the things we've discarded and the things that we're holding dear to ourselves as things we won't want to leave our culture. But nothing's rigid forever. And that's the important thing. That's the exciting thing. Seems to be a different company month on month. And that's wonderful. We're growing towards a goal and doing good things. And it's continually exciting. Thank you so much for listening to the first part of my interview with Sam Lambert, the director of systems at GitHub. If you don't want to miss out on the second part of the interview, which will come out in just two days from now, make sure you go ahead and subscribe to Developer Tea and whatever podcasting app you use. You can do that in iTunes. You can do it very soon. You'll be able to do it in Spotify. Hopefully, some of you listening now, you are listening through Spotify. So check and see if Developer Tea is in your Spotify. Feed. It's called shows in Spotify instead of instead of podcasts. But any podcasting app you use pretty much you can subscribe to Developer Tea. So make sure you subscribe if you don't want to miss out on future episodes. Thank you once again to Hired for sponsoring Developer Tea. If you are looking for a job and you're a designer or you are a developer, regardless if you want a contract job or a full time opportunity, Hired is a fantastic place to start. You don't have anything to lose because you can get on there for free and view interview requests. And on top of that, if you end up getting a job through Hired, they give you a $2,000 bonus. Of course, that is the special bonus for Developer Tea listeners. You're going to want to go to Hired.com slash Developer Tea. And as long as you sign up through that link, that bonus that is normally $1,000 doubles to $2,000. So again, thank you to Hired for sponsoring today's episode. And thank you for listening to Developer Tea. As always, you can reach out to me at developertea at gmail.com or you can reach me on Twitter at at developer tea. And until next time, enjoy your tea. See you soon. See you soon. See you soon.