Security engineering is more efficient
Applying engineering to security at a small scale can compound
Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer or any other entities with which I am affiliated.
I’m still hiring on my team at Headway. If you’re a security person who’s interested in working with a team with strong software engineering tendencies, please reach out to me or apply here.
That was a good segue into the topic of the newsletter today. I’ve brought up numerous times how there needs to be more engineering in security. The industry is currently too focused on risk operations, such as audits. In the past, I’ve written about this.
Why am I pushing for more engineering? Let’s first look at how security investments are currently going. It’s shown that companies that invest in security substantially reduce their chances of a data breach. However, data breaches are on the rise even as more companies are investing more in their security programs. That seems counterintuitive, but it isn’t shocking. The data shows that some investment is better than no investment. However, increased investment doesn’t mean they are good investments, which might indicate diminishing returns. Another important nuance is that having some investment in security does make it easier to detect and respond to security events and incidents, so it might mitigate or prevent the escalation of a security event into a more serious incident. Overall, my opinion is that many executive teams and security leaders struggle to understand and predict the risk in their organization.
So, why is this happening? Historically, risks haven’t been as technology-focused as they are now. They have been in many areas, such as IT, finance, legal, etc., but many companies have started to invest more in technology because that has led to efficiency and productivity gains. Every company is claiming to be and likely will be a software company. This means that most security risks will be technology-related, and unfortunately, adapting to this change has been challenging for security. It’s hard, especially since most CISOs don’t come from technical backgrounds — most used to do consulting or auditing. These technologies are evolving and being adapted at such a fast pace. It’s even hard for software engineers to keep up! I can only imagine how difficult it is for CISOs, and this is a large cause of stress and burnout for CISOs.
This article summarizes the core part of the CISO challenge:
Effective communication with the board of directors adds another layer of complexity, as CISOs must translate technical risks into understandable business impacts, overcoming varying levels of cybersecurity literacy among board members. Failure to do so compounds existing challenges, such as inadequate budgets and talent shortages.
With technologies, such as AI and cloud, many times, it’s hard for the CISO to understand the technology, identify the risks, and then translate them to the board. It’s hard enough to do these things individually. Honestly, it’s hard for engineering leaders to do this from a technical standpoint, and if the CISO doesn’t have a strong technical background, learning this under time pressure is increasingly hard, especially at startups where business needs evolve.
I’ve written about having only security operations is ok.
However, the issue is that it’s highly inefficient. This has worked in the past when executive teams and boards were willing to spend and invest in security, but as companies look to improve efficiency (and earnings), they will demand better metrics from security.
Why are security operations inefficient? Take the following scenario: Say that security discovers a vulnerability in the code. A good security operations team can likely reason about the vulnerability abstractly. However, they will need to pull in the proper engineering teams, give them context, and then ask the engineering team to go fix it. Now, the engineering leader is stuck weighing the business risk of spending resources to fix the issue with the security risk. This is similar to many legal functions at companies that present risks but aren’t enabled to fix them. This scenario results in a lot of back and forth between engineering and security to both understand sufficient context so that the two sides can align on the fix. Although it sounds simple, it typically is a lengthy battle where each side spends wastes a lot of time talking past each other without understanding the risk. A core part of the issue is that these teams tend to speak “different” languages — systems are complex, and technical context is hard to transfer. As a result, solving a vulnerability takes substantial resources, so security has to pick its battles because it isn’t an agent of change. This makes it difficult for security to surface vulnerabilities. Without technical context, it’s hard to classify a risk. Risks that might seem innocuous could be severe, but severe risks might already have mitigations.
How can engineering help, specifically a security engineering organization? A strong security engineering organization has software developers who can get technical context about the systems.
A common argument is that security shouldn’t touch code because the engineering teams with the most context are the best people to touch the code. The idea is that security doesn’t have enough context to write high-quality code. I strongly disagree with this statement. It’s an excuse used by security folks to not learn software development, and it’s indicative of someone who doesn’t understand software development. Any software developer can tell you that even the software engineers in the product and infrastructure have to gain context to write quality. In fact, many staff and principal engineers who do high-impact projects have to familiarize themselves with the codebase.
It’s very possible to run an effective security engineering team that contributes to the codebase. This is a point of high leverage for both the security and engineering teams. The security engineering team has a few advantages. First, they can gain both technical and security context so that they can understand what changes to prioritize and properly assess risk. Second, the team can be an agent of change. You don’t have to rely on engineering resources to solve problems. Security engineers themselves can make the code change and work with other teams. This might seem like a new mode of operation for security, but it is a common mode of operation for many engineering teams as most of the time, ownership is unclear. Overall, with the security engineering team, security can avoid the inefficient scenario above.
At first, security engineers might seem more expensive as they tend to demand both engineering and security premiums. If you look more closely, the security engineer can likely increase remediation velocity. They can also act in both a security and engineering velocity. This means that they can also help build tools to help better scale operations. For example, they can build tools to automate a lot of compliance work. Overall, this can substantially improve velocity as it doesn’t require coordination and negotiation between two teams where it’s hard to translate. Moreover, security engineering has the technical knowledge, so it’s easier for them to keep up
Assuming you have a mostly operational security team, how do you introduce security engineering? My recommendation is to hire a strong security engineering leader. This doesn’t have to be a manager. Many strong engineering ICs have strong leadership skills and can both do IC work and temporarily act in a managerial capacity. In fact, I would say some of the most efficient engineering teams have strong IC leaders.
It might seem that the surface area of the product and infrastructure might be large, and it might be daunting since software engineers will always outnumber security engineers. That’s ok because the goal of security is to reduce risk not completely eliminate it. Especially when starting, it’s best to allocate the senior security engineers to the areas of the codebase that are highest risk, such as authentication, authorization, web security, and infrastructure code. Another area to allocate security engineering resources is to have them focus on developing tools that can create a good interface between security and engineering so that security can get what it needs without asking engineering. This might require the help of a product manager, but many times, a product-minded engineer can drive most of this since the “customers” are internal.
It seems that having even a small security engineering team can reduce friction and create more efficiency in the security organization. It’s important to find security engineers who have a software development background. It’s also easy for them to understand new technologies and figure out ways to reduce risk. This poses a larger question. Will we start seeing more technical CISOs? I hope so. I think it’ll overall be better for the security industry, and maybe security will become more efficient and will build again!