Frankly Speaking 8/9/22 - Security needs more engineers

Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer or any other entities with which I am affiliated.

Quick plug for dbt! My team is looking to hire a senior manager to help build out the security engineering organization. If you’re interested or know anyone who’s interested, feel free to apply here and/or reach out to me with any questions.

LET’S BE FRANK

Well, the title says it all. As a security community, we need a stronger engineering mentality. In the past, I talked about my views on being a security engineer and its importance to a company. However, many organizations don’t have and/or misdefine the security engineer. My main frustration, which I will elaborate on more in this newsletter, is that most security organizations are too focused on justifying their value through work creation rather than value creation.

What do I mean by this? SOCs are a great example of this. I talked about how we should get rid of SOCs, and it represents a core problem that many security organizations are too operationally heavy and rely on adding people to solve problems. Work is defined by the amount of “manual” labor and time spent rather than the deliverable itself. Having people spend more time performing tasks provides unclear value to the organization. I would argue it only leads to additional operational overhead.

For example, in a SOC, you can add more analysts to look at tools or triage alerts, but it’s not clear if this is actually making the company more secure. Having more people look at alerts might be helpful, but that assumes 1. we are receiving the right alerts and 2. the analyst is actually interpreting the alert properly, which can vary depending on the experience and quality of the analyst. We need to better engineer incident detection and response. Adding people alone is not the solution, and I have described in my past post on SOCs how to better “engineer” incident detection and response.

Let’s take a step back. In most organizations, problems are solved through a mixture of three levers: people, process, and tools. For an engineering issue, we work on creating/buying tools and processes to solve the problem. People come in the loop to build or buy these tools and define these processes. However, the goal is always to automate this work and “move up the stack,” meaning the goal of the people is to build not to operate. This goal makes it possible to scale people without increasing headcount. DevOps, despite having “Ops” in its name, has done a good job building tools to automate infrastructure, taking away much of the operational work that was done traditionally by IT.

Security has tried to follow a similar model as many parts have moved from IT over to engineering. However, even in companies with heavy cloud usage, this move has happened more slowly. In part, organization leadership is to blame. Many engineering leaders don’t understand or know how to better “engineer” security like they do with DevOps, so the incentives and responsibilities for security are regularly misaligned with the goals of the company.

That’s why there is no strong correlation between money spent on a security program and the likelihood of a breach. In fact, there are very few studies done on this, and there are more studies done on the cost of a security tool vs. the cost of a breach. This is one of the least productive comparisons and has resulted in the mentality that security problems can be solved solely with tools rather than treating them like an engineering problem where a solution involves a mixture of tools, people, and processes.

As a result, security products have flooded the market. I have talked about why (almost) all security products suck. Most of these tools create “false work.” They generate alerts to demonstrate value, especially to a security team that lacks engineering context. In order to “properly configure” and “triage the alerts,” security leaders ask for more headcount to help manage the tool, which is antithetical to the engineering mindset of scaling. I put quotes around those phrases because it’s not clear doing this will lead to better security.

However, don’t blame the security team! Blame the tools and the leaders. Security leaders need to stop buying bad tools that don’t solve real security issues, and company leaders need to hold security leaders to the same standard as other engineering organizations, i.e. build processes and tools that scale. Also, by building tools and processes in security that allow for scalability, an organization can alleviate the common complaint that there’s a talent gap in security by allowing existing security talent to scale and focus on solving other problems.

With all that said, the solution is simple but complicated to implement because it requires a mindset change. Security needs more engineers, who are focused on building solutions rather than operating tools. It’s ironic because attackers have figured out how to build tools that easily automate their attacks, but we haven’t found good security tools to automate our defense against them.

This solution is also complicated by the fact that engineering-focused security leaders and security engineers are hard to hire. Engineers are in high demand with limited supply, and security-focused engineers are in even higher demand with even more limited supply. Moreover, for a long time, company executives have been hiring security leaders that have been focused on addressing security risks through creating operationally heavy organizations. There aren’t many security leaders who have experience building out engineering-focused security organizations. So, it’s hard to address to fix this problem when talent is so limited.

It’s frustrating for me to hear people describe security as a cat and mouse game between attackers and defenders. You would never describe any other engineering function like this. No one describes bugs in code like this, i.e. it’s a battle between finding and fixing bugs in code. So, why would you describe security like this?

Security has suffered too long as an organization where value is derived by operating rather than building. In order to have an effective security organization, we need to adopt more engineering principles, such as defining problems and engineering solutions, rather than blindly buying tools that require substantial operational overhead.

Here are some open questions:

• What can we do to get more software developers interested in security?

• How do we, as a security community, help engineering leaders construct better security organizations?

• How do we improve existing security tools?