At WayScript, we spend a lot of time thinking about developers and the challenges they face. Today, we’re focusing on productivity in the org. We wanted to hear from an expert who spends a lot of time thinking about this challenge. This week, we were lucky enough to speak with Kathryn Koehler, Director of Developer Productivity Engineering at Netflix.
About Kathryn
Kathryn is an engineering leader with over 25 years of experience in software, the majority of which she spent writing code and managing engineering teams. She’s currently the Director of Developer Productivity Engineering at Netflix, focusing on making the everyday lives of Netflix developers easier by providing the best possible experience for local development. Before this, she led engineering for the Science Initiative at Chan Zuckerberg, where her teams built tools and platforms for biomedical researchers with the goal of accelerating scientific discovery. Kathryn also serves as a mentor and coach to underrepresented people within the tech industry.
Full Interview
Interview Transcript
Monica from Team WayScript
At WayScript, we spend a lot of time thinking about developers and the challenges they face. Today, we're focusing on productivity, which is a buzzword we hear a lot, but it's a crucial component for successful dev teams. We wanted to hear from an expert who spent a lot of time thinking about this challenge. So we were lucky enough to have Kathryn Koehler with us today to talk about how development teams can actually engineer productivity. Welcome, Kathryn.
Kathryn Koehler
Thank you for having me.
Team WayScript
What is the role of a productivity engineer? How does it differ from a traditional SWE role?
KK
Productivity engineers build tools and infrastructure platforms that your more customer-facing engineers develop on top of. I've said this a couple of times in other podcasts, but we're kind of a nerd’s nerd. And I think that we sit in a very unique position centrally within the organization where our customers are our internal developers.
We have a very close knit feedback loop. We're also customers of the stuff that we produce, so we're dogfooding our things every day and as a result, we know how developers think. We don't lower the bar in terms of overall customer experience in terms of UI and things like that on our tooling, although we do fall victim a little bit to dev art here and there. And we rely pretty heavily on API's and the command line interface, of course, but some are a little further down in the stack than then most folks usually operate. And we have a very close partnership with our cloud infrastructure folks and our data platform folks. So that's where we sit. Yeah, we are different. But, but cool.
Team WayScript
And important. Productivity can be a loaded buzzword with multiple interpretations. How do you think about productivity at Netflix?
KK
We are that higher leverage point. So if you start using something over and over and over again, or it becomes an “ism” of the organization, for ex, “this is how we do security”, “this is how we do performance optimization”, “ this is how we connect with our services”, we start to pave out those general use cases and tooling so that folks don't have to do that over and over and over again.
If you're new to Netflix, or if you're new to an organization, you can bootstrap and get up to speed and start contributing code in your particular domain very quickly, rather than having to write a bunch of boilerplate code, having to think about a bunch of considerations that aren't something that you're an expert in. So we do a lot of that abstraction. And we hide away those implementation details which makes sure that you can still develop easily within the Netflix ecosystem.
Team WayScript
There's a lot of focus there on the experience of being a developer at Netflix and being able to spin up and start your applications quickly, producing production code and impacting the customer. How much of productivity do you think is about the experience or qualitative feeling of working on the team?
KK
It goes very hand in hand with each other, it's very interconnected. The developer experience if your tools are not enjoyable to use, if you have not created the right abstractions, if people are still bogged down in the muck and mire of implementation details where they have to reach out to help, where they have to search through a thousand different varieties of documentation that may or may not exist, that's a terrible experience. If we can keep that as frictionless as possible, then people can really focus on their day job.
And so for our customers, that could be anywhere from front end development, back end development to edge, etcetera. We want them to be able to provide as much impact for their customers, which are our shareholders and subscribers as possible. Subscribers, our customers, shareholders reap the benefits of that. So if we are good at our jobs, then we're a leverage point where our customers, Netflix developers, can have a tremendous impact on the work that they're doing.
I've heard some folks say a productivity engineer is worth 10 engineers and I don't buy into that crap. I mean, that's ridiculous. That's a lot of hubris. But if we're doing our jobs right, then we can save developers so much time. And if they're spending their time wisely, which they hopefully are, and because everybody likes to be productive and successful, then they're focused 90% of their day on delighting the customer, as opposed to maybe 25% of the day, because they're constantly wrangling tooling that doesn't place play nice with each other.
So I think they go hand in hand, the actual developer experience and the kind of impact that we can have with our customer in moving the needle, to throw out some manager terms.
Team WayScript
Yes, you got to have that jargon in there. (laughs)
KK
“Laser focused bias for action always”!
Team WayScript
It sounds from an organizational point of view, having your team be the centralized team that works across all of the development orgs helps with that discoverability of those tools, as well as the standardization or the automation of those “isms” as you were talking about?
KK
Yeah, absolutely.
Team WayScript
Speaking of the diversity of your user base, of course each team has their own use case, their own frameworks, their own languages and tools. And so you're creating certain “paved roads”, as I've heard you call them, but they don't necessarily apply to all teams. How do you think about balancing these a la carte requests, if you will, with creating more of a generalized common engineering experience?
KK
Yeah, that's a really good question. We are sort of the victims of our own success in terms of tooling proliferation. We look like a pretty busy map of a nice metropolitan city without a ton of highways. And not that highways are great from a neighborhood planning perspective. But in terms of the paved road, or paved path concept, we need to get more people onto the superhighway that has them be able to get from one place to the next pretty easily. And right now, unfortunately, we still have a lot of side streets and some streets that don't go anywhere.
Team WayScript
Or only ones only the developer knows about.
KK
Exactly, extra secret streets. So we are going through the process of paving more with water permeable paving that's good for the planet and paving more for our users connecting those experiences together.
We have two primary languages that we leverage at Netflix, and that's Node and Java. We don't have so much paving around the other languages. We want to make sure that we have a multi-language story so that those folks can get the support that they need and around build, package, publish, security, performance, optimization, etc. But for the two frameworks in which we are most focused right now, we still are paving and connecting the experiences so that you can come in and have a seamless “build it, bootstrap it, build it, test it, deliver it, manage it when it's out the door”.
That is really tricky. Because once you've grown up and you let the cat out of the bag and you have all these disparate experiences, how do you tie it together so it doesn't feel like a Franken solution? We're still going through that right now. That means more curation, more abstractions. And you mentioned this earlier in your question, we really should be going off of use case driven work.
So let's take a persona, let's take a specific customer, what are the use cases that they need? And let's thread the needle all the way through the experience and make sure that that's a good paved experience, because otherwise, you could end up going to 80% on everybody and it still sucks, right? Take those high leverage business critical use cases, start paving them thoroughly, comprehensively, and then approach it that way. Over time, and with a lot of sweat equity, you'll build out the true paved path experience.
You mentioned “how do you balance a la carte versus whatever”. I love really bad analogies. It's sort of my Hallmark. And so what I think a lot of productivity groups end up with is sort of the Cheesecake Factory experience where you have a thousand page menu that's laminated, and it's got all these different cuisines. And you know, I'm sorry, Cheesecake Factory, but it's a novel to read through. No matter what you order, it's not going to be that great. But if you can pare down those offerings, and think of a really nice dining experience where the menu is a la carte, or you could go prefix if you want to, but it's all in a specific type of cuisine. So no matter what you order off of the menu, you can share it and it'll work well with others. And then think of your productivity engineers as the service: you don't want to have to interact with us too much, because that means that something was wrong with the meal, something's wrong with the bill. Something's off. So you don't want to have to reach out to us, you just want to have a really nice experience, and leave that restaurant feeling satisfied.
Team WayScript
Like a guided experience.
KK
Exactly, a guided experience. We can give you the wine pairing, right? If you want to, we can let you pick, but we're not going to give you something crazy or we're not gonna have you walking to the back of the store room to look at the different wine options. We're not going to send you to the farm, pick it yourself and bring it like “we need you to go kill this animal yourself”.
So that's how I think about the whole experience and tying those experiences together, being very thoughtful about what we go after. And those use cases that we're delivering on.
Team WayScript
Especially when it comes to dev standards and how to do best practices or how to build the right process, so much of it is opinion driven. And it's “well, this is how I've always done it, therefore it is best practice”. But when you ask “does this truly solve for my desired end state?”, the answer's no or not completely?
KK
Yeah, absolutely. Or, say you're solving it for the super user. And this is a trap that we fall into within productivity within central engineering. Because our customers are engineers, they're going to be very technical. We're engineers, we let them config the heck out of everything. And so everything's exposed, like “go for it, we give you the world”.
But really, sometimes, why not curate it for that 85% case, but then give people access to the configurations, things that they need for that 15% that you can't necessarily cover? That’s another thing to really think about. How do you balance that super user versus the everyday user? Fine grained control etc?
Team WayScript
I agree. And I think that there's something to be said about the ability to have it all in one spot too. You can have more eyes on it than just if you were doing it on yourself or if you're just duct taping something together.
KK
Yeah. And also when you have a thousand different solutions, search and discovery on those is a nightmare. How do you even figure out what the right tool for the job is? Gotta keep it simple!
Team WayScript
Yeah, exactly. And you're like, “I want my infra person to tell me how to think about security in here”. Or I want them to figure out the really hard things and to have a platform like the one that it sounds like you've built at Netflix, that allows engineers to just experiment with some of these processes to write and create POCs that much faster than setting up the environments from scratch.
KK
Exactly. It's the separation of concerns, right? It's the ultimate form of object oriented development, not to nerd out too much.
Team WayScript
Oh this is the spot to do it. Once we've created these tools and paved roads, is user adoption a challenge for you? How do you make sure that within Netflix itself, people are using it?
KK
Yeah, it is a real challenge. And we don't have a very top down corporate culture where we have folks higher up saying “you must do this, you must do this”. This is the paved road or the highway. So we need to make sure that when we are building things, the story and the reality is compelling. If you move on to this, if you adopt this, if you migrate to this latest version, or this newest thing, what does that get you?
Luckily, I think a lot of companies went through log4js remediation at the beginning of the year, which was this Java exploit. And for folks within Netflix who were on the paved path, it was a lot easier to deal with.
I think that's a really compelling argument for folks. But you know, for other folks who are just fine using what they're using, they're not going to get a ton of the benefits that we provide within productivity engineering. But also productivity engineering is going to be on the hook to support those things, even if they're not something that we are actively developing. So it's in our best interest to make sure that folks have a great experience when they're using our stuff. It's within our best interest to make sure that we're aligned with the customer's needs so that we build things that they really want and need. That’s tricky business, because people are really good at saying what they don't like, but not what they really need.
We have a product org, we're product driven within productivity that helps us really align with the customer's needs to make sure that the loudest voice doesn't win, that it's the highest leverage and the highest impact thing that we can go after. Strategically, that gets our focus. And that's very helpful to stay super close to what the customer needs long term.
Team WayScript
When you think about measuring the impact of developer productivity on these kinds of initiatives, in the past, I've heard you talk about measuring outcome over output. Could you touch on that and explain why it’s important?
KK
It’s easy to get into the activity trap. As a developer, I'm pushing a lot of code. I'm a hands on keyboard all the time, I'm just cranking stuff out, boom! And so what I mentioned earlier, if you're not actually lined up with what the customer needs, it doesn't matter how much code you're pushing. That's fantastic. In some cases, like you could be writing more lines of code, because you're sort of floundering, right. So I don't want to see activity. And although activity is a great signal, it could go either way.
So I want to make sure that we're measuring output. What are those key performance indicators? What are those customer hypotheses that you have around pain points? And how are we solving them? And then making sure that we're circling back with the customer, either, through our own instrumentation and metrics, or through qualitative feedback: How's it going? That thing you were really upset about or didn't work last time, how's it going now? Do you find that you're spending more of your time working on the things that you're passionate about?
We can take away a lot of the “isms”, right? You could work very smartly on the thing that is highly aligned with the customer and not be in the office for 15 hours? Right? Or maybe the opposite! And be like “What’s going on dude? Where are you?”
So being very clear on that front and also incentivizing outcome and not output. When you're taking a look at your teams and contributions and performance, don't have the laundry list of things they shipped. Have the laundry list of how the thing they shipped impacted the customer. And I think that's really important, because what you what you value as a culture on the team is what gets rewarded.
Team WayScript
100%. And it takes productivity from being this competitive benchmark to compare people against each other to “Oh man, this is actually useful for us and for you as an individual and for the company as a whole”.
KK
Yeah. And one of the things that I love about Netflix–and there are a lot of things like the culture that is very strong–first and foremost, it's what's the best thing for the business? Think beyond yourself, think beyond your team, what is the best thing for the business? What is going to move the needle there? And that really helps clarify and prioritize the work that we do.
Team WayScript
To wrap up, any advice that you would have for people starting out in dev productivity, especially those who are new to it all?
KK
Yes. Ah, gosh. Before you get too big, don’t build everything from scratch. Buy things off the shelf and tailor them. We’re not as special as we think we are: we become very special and bespoke the bigger we get, the further away we drift from infrastructure and things that are available. And it's not saying don't have a productivity team. Productivity teams are awesome. But be very smart about what you build versus what you buy and lean into buying as much as possible, because you don't want to be holding the bag on a lot of things that have just drifted so much from what's now industry standard, that you're stuck in that position asking, “Oh, do we tear it all down and go back to what everybody else is doing?”
Really figure out where you want to distinguish yourself in terms of business logic and IP and things like that. And that's what you build. All the other stuff, there's so many great companies out there that are doing this stuff for you.
Team WayScript
Like building source management.
KK
No, don't do it! Gosh, if you had the time machine to go back and be like, “oh, man, we shoulda just put the money down on that early.” Leaning into open source, leaning into more enterprise solutions where you can, can really have a huge impact on the amount of productivity teams can have from the beginning.
Team WayScript
Well, that is wonderful advice. Thank you so much for joining us today. Kathryn!
KK
You're so welcome. Thank you.
