How to be a security person that engineers don't hate
Repost of a popular past article
Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer or any other entities with which I am affiliated.
I was traveling for work last week and preparing for RSA, so that’s why this newsletter is coming out late and is a repost.
I wrote this article about a year ago, and in my opinion, it’s more important now than ever. It doesn’t just apply to partnering with engineering teams, but it will also start applying to data teams as we continue to find more applications for AI. As security threats become more technology-focused, security teams need to be seen as key collaborators helping to drive business outcomes rather than limiting them. This might be tough to develop, especially since security has behaved defensively for so long. However, I do think this is necessary not only to help prevent threats but also for security teams to stay relevant. Inevitably, these teams will have to adapt.
I recently came across these two articles, “How to be a PM that engineers don’t hate” and “How to be an engineer that PMs don’t hate,” and was inspired to write something similar for security and engineering. There’s always talk about friction between these two organizations. At some companies, it’s friction between the DevOps team and the infrastructure security team, and at other companies, it’s friction between the product teams and the application security team. Many times, it’s both — it’s a tale as old as time. It’s so commonplace now that many security leaders have chosen to accept and operate under this status quo. That is, security leaders feel like it’s part of their job to complain about their engineering counterparts. In fact, sometimes security leaders feel “lost” when they don’t have this friction because they feel that most of their job is navigating this tension.
The main theme of my blog is to increase collaboration between security and engineering because this friction is counterproductive and bad for business. However, most of what I discuss is at a business or organizational level, specifically on how to build a security engineering organization or how to introduce engineering principles into security organizations. Recently, I wrote that having just security operations is sufficient. To go further down this path, in this newsletter, I describe how security can collaborate better with engineering (without actually having security engineering as a bridge). We can break unnecessary this cycle of tension without drastic organizational changes.
Like the articles discussed above, I am proud of the bridges I’ve built in my career between security and engineering. Having less friction is always beneficial for an organization because the bandwidth wasted on solving this friction can be spent on solving more impactful issues.
So… how can you be a security person that engineers don’t hate?
Inevitably, there will always be friction between two organizations that have slightly different goals, but it doesn’t mean they can’t work together effectively. I view security as an internal service, which means that we create a product for our internal teams to consume. Therefore, a key goal of security is to be a good partner to other teams. Here are some ways to be a good partner:
Understand engineering processes
Gain an understanding of the product and the business
Demonstrate how solving your problems can also achieve their objectives
Shadow and work with the team
Understand the engineering processes
Sprints. End-to-end testing. Git version controls. Pull request reviews. Terraform applies. These are engineering-specific processes and tools, and they most likely don’t exist on operational teams. Unfortunately, this is a tried and true process for engineering, and many teams abide by it. It exists for a number of reasons that I won’t go into now, but its goal is to create stability as many people build on the same platform and application. Imagine you’re building a skyscraper without any work processes. Someone might actually dig where there’s an electrical line. Too many people will be working on the same part of the building, but other parts remain underdeveloped. It’s not exactly the same, but it’s a similar idea.
What this means for security is that seemingly simple fixes might take longer than they imagine. For example, upgrading a package is not as simple as changing a number in a file. Similarly, on the infrastructure side, making a configuration change isn’t as simple as changing a value in the AWS dashboard. It has to change in Terraform and undergo tests and approvals. The reason for all this is that changing a value in a shared application might have unintended side effects, and engineering needs to make sure that these side effects don’t take down the application, thus creating a bad experience for the end user.
It’s important for security to understand these processes so that they understand the workloads of various engineering teams and know how long a fix might take so that they can better prioritize their asks. On my team, we used to follow sprints even for operational work so that we can better time our asks and also get a feel for engineering processes as they also vary from company to company.
Gain an understanding of the product and the business
Let’s be honest here. Most security teams operate on an island. They think about security problems in isolation, usually with very little context for the product and business. However, in recent times, this has started to evolve as product and application teams start asking security for advice, forcing security to care more about the business.
Security leaders should learn more about the product/business and force their employees to do the same. This helps in numerous ways:
Security can better tie their work to business objectives and thus ask for resources more effectively rather than using FUD-like techniques
Have better context on business risks so that they can better prioritize security fixes
Can deliver value through security fixes that improve the product for the end user
The list goes on. Ultimately, security teams need to understand business risk and balance it with security risk. Sometimes, it’s necessary to take a security risk for the business because all businesses have to take risks. Part of the job is to find and understand the security risk so that appropriate solutions can be created. This is a nuanced process that security leaders should avoid reducing down to traditional FUD, i.e. “we will be hacked if we don’t fix X.” Without a business, there’s nothing for security to protect!
Demonstrate how solving your problems can also achieve their objectives
Similar to understanding the product and business, security teams can collaborate with engineering teams to create urgency around projects. Many engineering teams, similar to security teams, are under-resourced and overworked. They have long backlogs that they are struggling to take on. Every planning cycle is a battle around priorities. Security teams can surface issues and rather than creating friction, they can be a partner where creating a joint solution can solve both teams’ problems. This will also make the problem more appealing as it doesn’t require just one team’s resources because security can also provide resources in this partnership.
For example, upgrading packages and dependencies can be sold as part of an initiative for platform stability, which can benefit platform engineering. Similarly, improving developer security through better access management can also improve developer experience, which is likely an engineering-wide initiative. These are just some examples, but solving security issues or building processes around them can, many times, also lead to better engineering velocity and operational excellence. It’s just a matter of how they are being messaged. Approaching security issues in this way can create a partnership rather than friction.
Shadow and work with the team
Finally, other platform teams typically shadow the team they support, so it’s also important for security to do this. At the very least, security can build relationships. They can also offer to get their hands dirty. This way, they can understand the team and their workflows. Sure, security might not be able to do the engineering work, but they can help with the operational work, such as testing and documenting. Just attending their standups and sprints can give a sense of the problems the team is facing. It improves security’s ability to gather context and present solutions to the team rather than just reporting vulnerabilities and issues, leaving the solutions in the hands of the engineering team. By working alongside the team, security becomes perceived more like a partner rather than an auditor.
Takeaway
Friction is common between engineering and security, but it doesn’t have to be. An organization doesn’t need a security engineering team to break the cycle of friction. There are basic steps security can take to ensure they behave as a good partner rather than an auditor. This way, security can be perceived as solution creators rather than decision makers/imposers.