At WayScript, we spend a lot of time thinking about developers and the challenges they face.
Today, we’re focusing on security which is critical to software teams, regardless of size. Security vulnerabilities come from the code we write, packages we’re using, any open-source code we use, the hardware, even the business logic. It can be overwhelming to address them at an organizational level, especially if you’re building the team.
We wanted to hear from an expert who spends a lot of time thinking about this challenge. This week, we spoke with Tyler Healy, VP of Security at DigitalOcean.
About Tyler
Tyler is the VP of Security at DigitalOcean, a cloud infrastructure provider headquartered in New York, where he’s been for four years. He has over 16 years of experience in Information Security working across industries in defense & federal, telecom, and pharma. Tyler is a Fellow at the Institute for Critical Infrastructure Technology, a D.C.-based non-profit for cybersecurity research and policy and an advisor for WayScript.
Full Interview
Interview Transcript
Monica from Team WayScript
Welcome, Tyler! Starting off, can you tell us a little bit about what a security engineer is? And how is that different from a software engineer?
Tyler Healy
Thanks for having me! There are actually a few different flavors of security engineering. To bring it to the DigitalOcean side of the world, we have a security software engineering team, who are the more traditional software engineers, but who are building security services for the rest of our engineering teams to use. That could be secrets management encryption libraries or fraud and abuse detection services that run DO. They’re building offensively for customer facing or customer impacting services. They are more traditional software engineers.
But there is also a brand of security engineer, I think it's a little more traditional security engineer, where it's an engineer who focuses on building or integrating security products into the development environment or the company as a whole. Identity and access management, vulnerability management, securing your laptop–there are hundreds or thousands of security tools that a company could integrate into their environment.
There is a whole world of expertise around security engineering and how to engineer a company and the company infrastructure securely. That can have endpoint focus or network focus or identity focus. You can get incredibly deep into each one of these different verticals.
And so when you say security engineer, it's actually sometimes a little difficult even when we go through the hiring process, because we could say security engineer, but it could mean one of twenty different focus tracks for people. It’s interesting, because there's so much to get into in the security space, that being a security engineer can open so many different doors to different layers of expertise.
Team WayScript
How does an organization determine which of those verticals they want to invest in?
TH
A lot of that ties back to the risk landscape and kind of the threat landscape for the business and the different sides of the business. A startup that has fifteen people probably doesn't need someone whose entire job is focused on vulnerability management, or even identity and access management. You're gonna bring in someone who can cover a lot of those different topics.
From a broad standpoint, it starts with the risk landscape. So if you're a FinTech company or crypto exchange, if you have business customers versus consumers as your target market, if you're purely a technology company versus a tech enabled services company. All of those things will define where you want to really start, and a lot of it just comes back to the customer.
How are we building trust with our customers? That's usually the best starting point is to say, what's the customer data? How are they interacting with our product? And then that can start to lead you down the path towards where your first real investments should be from a security standpoint.
Team WayScript
You have a team that's integrating security products internally or creating tools for engineering teams at DigitalOcean. What does that interaction look like? How do they fit into the org as a whole?
TH
The model that I believe is essential for security teams is a relationship led partner based model with the rest of the business. There's a traditional view of the security world: security writes a bunch of policies and then tries to apply it to the business. It fails miserably a lot of the time because they're not in it with the rest of the organization learning about what their problems are, and how to solve them together.
For example, at DigitalOcean, we have a lot of systems engineers because we do a lot of kernel work and performance work. There's a big difference for our customers and for us to business how much performance we can get out of each bare metal hypervisor. And so we have teams dedicated to that space. Securing the endpoint devices that folks are working on in that space where you can have that little bit of performance impact on the engineer makes a big difference to their day-to-day job.
It’s important to get into the mindset that you're working together to secure the business relative to what everyone needs to do for the day to day job versus dropping a security control framework. Because what you end up doing is you end up having people work around you. That introduces more risk to the business. You want to be out there, be a part of the problem solving, and get friendly with everyone in the organization. It opens lines of communications where you can hear things that might be risky and get on top of it quickly.
Team WayScript
This reminds me of something I've heard you say before, which is that security is a team sport. And that collaborative approach to security is essential for it to be adopted. What does that look like at DigitalOcean on a day to day basis?
TH
That's a great question. It’s where I've spent a lot of my security career over the past half decade: thinking about how security teams are successful and less about the technologies themselves. Technologies will come and go; the threat landscape is always changing. One of the best parts about security is that there's never a dull moment in security. Something is always happening and changing.
But just being really good at security technology is not what makes a security team successful. What makes a team successful is the ability to be engaged across the business.
And so for my day to day, I probably spend more time with marketing and finance and the product teams than I do even with the technical teams sometimes. And a lot of that is just being in the conversations about where we're heading as a business: Where do we think our customer base is heading? How are we building trust with our customers? All of that starts way before you start picking security technologies.
An example for DO where security is really plugged into the business, not just trying to secure the technology that the business is building, is we own all of the fraud and abuse tooling that happens during a customer signup lifecycle. Say someone comes to the DigitalOcean website to create an account. Cloud infrastructure can be used to do fraudulent and harmful things. Some people try to create accounts in large volumes to ostensibly defraud us or other people or harm others on the internet. It’s our job to help secure that. So we are at almost every single stage of the customer signup lifecycle. Marketing is also invested in this lifecycle because they're trying to drive more customers to our platform. We're striking a balance between friction through the customer signup lifecycle–making sure their engagement is smooth and we're not adding too much friction–and preventing the bad guys from using our platform to do harmful things.
That natural tension between security and marketing I have found to be one of my strongest relationships. We have such a strong working relationship where we get to talk about the real problem of security and how that interacts with the growth targets that we have. Especially now as a public company, those are public growth targets which investors are looking at. We are constantly building the right tension and turning the levers in the right way to accomplish both goals. It's been rewarding to be in that part of the conversation instead of just “alright, someone’s building something and we got to help them figure out how to go.”
Team WayScript
It doesn't feel like an obligation to bring in security at a certain step.
TH
Yeah, exactly. And that's where you want to be. The business logic conversation is a lot of where their security problems manifest from the very beginning. If you're bringing in security super late to plug the holes after the ship is already sailing, it's not going to be a great situation. But if you can be a part of that that ideation process the business logic that we're trying to bring to life, you can do it in a way that can sustain the business rather than than hampering it.
Team WayScript
That’s a very important take and perspective on security: it's not meant to slow down velocity or slow down innovation, rather enable it. What does DigitalOcean experience now that security is so involved in the conversation?
TH
Yeah, yeah, it's great, because we get to be part of the product release lifecycle. We’re bringing a new product to life and a new product to our customers. We're about to launch our serverless offering, which is pretty exciting. We had a dedicated security resource from day one on that launch.
And we uncovered a few things early on, that were going to be a little bit of a concern by the time we got to GA, and we were able to work with product leadership and engineering leadership to make some adjustments along the way to to address some of those risks before launch in a way that would allow us to have a both a successful launch on time. But also, one where our customers can trust the product right out of the gate. And so it's hard to do, because sometimes you do have to make concessions, like, there are tons of features that do products have launched within the past, even while I've been here, where it's been man, I would really love to have this particular security feature on the product, by the time we will. But that's not always possible, because GA dates have a lot wrapped up in them in terms of customer expectations, marketing expectations, investor expectations, and you can't really miss those marks. And so it's about figuring out what is the right tradeoff to be making. And you can only really do that by having that team sport, early engagement mentality and building that culture within the security team.
Team WayScript
Being there in the moment, right? Instead of getting caught up afterwards. When there’s a GA launch, there's so much risk about that timeline moving and I think people will often view QA, compliance, security, as what adds to that risk factor.
TH
We get to have open and honest conversations about what those things mean and make the right call. It’s a risk based decision because we could be adding risk to ourselves or to our customers. And in some circumstances, that's going to be okay, there's no such thing unless you're a particular government agency, there's really no such thing as a zero risk game. You're always taking in some amount of risk, especially as a cloud service provider, or anything where you're a developer experience environment where you're handling someone else's computing power, really no matter what product you have, you set some amount of risk baked baked in, and it's about making the right choices at the right time.
Team WayScript
How do you make sure people off of the security teams hold up their end of the bargain? How do you make sure Marketing or Sales includes you in those conversations?
TH
Yeah, that's always a tough thing. But it starts very early on in a security team's life cycle, and it changes though during the size with the growth stage of a business. So when a company is like 50 people or 100 people, or even up to like really 250 people, you can make it very relationship based where you spend a lot of time talking with various business leaders in all the different parts of the organization. You communicate the goals of the security program in a way that people can attach to and so much of it comes down to communication.
That's a big piece that sometimes gets lost in security is just how important communication is for the security team to connect with the rest of the business which then helps you achieve your goals.
So that's how it starts, but eventually you know if you get to a 1000, 2000, 5000 person company or even larger than that, you can't know everyone in the business and you certainly can’t implicitly trust everyone in the business is going to be doing the right things. Because humans are the weakest link in security, that's just a fact of life. What you end up having to do is start to institute stricter controls and have a broad based policy approach in a way that is connective with the organization at large.
But unfortunately, you can't just be all places all the time. So I suggest teams create a “security advocates model”. Some really large companies that I've seen in the past, do things like a security advocates model, which is like a business information security officer, where a particular line of business will have a dedicated person to security. Their whole job is essentially to be a partner between the main security team and that business unit. That only happens when you really start to get large scale. But you know, security advocates are a great way to get started if you don't have dedicated resources across each of the verticals.
Team WayScript
Say you are just starting off. And so your engineering team is like maybe twenty developers or something like that. If we haven't decided that it's time to invest in a dedicated security resource, then what should we be doing? Or should we be thinking about hiring that resource?
TH
Probably around fifty employees is when companies start thinking about a dedicated security person. At the very least that can be someone who is sort of hip to hip with the engineering team, depending on the type of product and the style of business, but that's generally where it would sit.
It comes down to fundamentals. Every engineer should have the idea of two factor authentication, not putting secrets in code, using encryption appropriately, and treating customer data appropriately. You don't have to get super fancy with all the tool sets. But you just have to get those basics down. And as long as you have those as a starting point, by the time you start to add a security team, life will be a lot easier to integrate in a security team to the organization.
Team WayScript
How do you minimize friction when you're working within DigitalOcean itself? Can you talk a little bit more about the perceived tension security teams have to fight against?
TH
There's a few different ones. One is that security is the stuff that's going to slow me down, right? Obviously, product releases. And again, that's just the a lot of that just comes down to messaging about how do we get involved early and make things less painful towards the end. I have experienced in the past where someone will go to do a product release and realize that they haven't gotten security involved at all. We've actually had to squash a release. It's been fairly minor, generally speaking, but it does happen. So there's tension there.
There's also the tension that security is watching me all the time, which is a really funny one, because we're really here to protect the business and our employees and our customers. But, you know, you deploy an agent on every single laptop, antivirus, anti malware purposes, whatever. And sometimes all this can see everything I'm doing, there's kind of a brother factor in all, that one's a little tougher to solve for, because you'll always have people in the organization to think that way. And that's okay. They're very paranoid about that, and it is certainly their right to think that way. I don't try and fight it, I just kind of embrace the fact that there will always be that kind of feel.
Team WayScript
Right, you have to acknowledge it and give as much information as we can.
TH
That's exactly right. Try to be as transparent and open about how it all works. And you know, when do we exercise administrative privileges that could see data and how we protect those things and just be really open and honest about it. At the end of the day, you'll never convince everyone.
Team WayScript
Any last thoughts or advice that you would give to maybe as someone who was in your shoes, but at a much smaller organization? Anything that you want to leave us with?
TH
It's been excellent. Thanks for having me on. I would just say that IT security is just as much about thinking about what the business needs as it is about security, solving security, and technical problems, which are super fun. Obviously, they're really exciting problems in the security space, but keeping the business frame framing in mind, is really a key to success,
Team WayScript
Love that. Thank you, Tyler!
TH
Thanks for having me!
