Breach simulation is a complicated market
It's not clear this service should be a product
Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer or any other entities with which I am affiliated.
We’re hiring a senior product leader for our trust (security and privacy) organization at Headway! If you’re interested in helping build a mental healthcare system that everyone can access, you can apply here or reach out to me.
What’s breach simulation?
There have been many companies in this space, such as AttackIQ, Cymulate, SafeBreach, and Praetorian. These products' “core” functionality is to simulate real-world attacks, but their marketing varies. Some say they will test the effectiveness of security controls and tools (Cymulate has moved more heavily in this direction). Some say they will help augment or replace red teams. Others have said they will help “manage your attack surface,” i.e. find potential attack vectors and understand the exposure. The high variance in marketing shows that this is a complicated and tough market.
The problem they are solving is unclear.
From one perspective, the problem is clear. They are generating attacks to test how effective your security program is. However, it’s not clear what exact business problem they are solving. Are they trying to reduce the size of red teams, or trying to replace them? Are they just a way for security teams to prove they are actually doing something by providing some metric? If so, why do they have to happen regularly rather than a few times a year? Is it necessary to prevent a breach? How do we measure the risk vs. the cost of having this tool? As a result, it’s hard to justify this type of tooling because it’s unclear what value it brings to the organization, let alone the business.
Another way to see the problem in this market is to understand the status quo. Before these products, security teams used services to find vulnerabilities or “soft spots” in their defenses that would lead to breaches. This was either in the form of a penetration test or a red team on an annual or biannual basis. As companies release more products, they usually ramp up how many times they would use external services to do this. At scale, it would be cheaper and more efficient for companies to have their own red team. The reason is that there’s a need for more frequent pentesting, and also having in-house pentesters allow them to be more targeted in their findings as well as have more business/engineering context to find flaws.
Turning a service into a product is hard.
In general, turning a security service into a product is hard. A few companies have done this, but they have managed to use their service to augment their product. The one example that comes to mind is Crowdstrike. They were doing incident detection and response, and they realized that the tool they were using most often was one around endpoints. However, this was an instance of a service using a particular tool and commercializing the tool. Then, they were able to augment that tool with their services. What also helped is that having more customers on the tool allowed them to collect the necessary analytics to make their EDR tool even more valuable.
However, this instance of pen-testing and attack simulation is slightly different. Although several tools are used, such as BURP and some other specific tools, there is a heavy manual aspect to achieving high quality. In fact, I believe the best pen testers are those who only lightly use tools but instead spend time with the target product to understand how it works. That’s why having an in-house team is an advantage, as it allows them to find more sophisticated bugs and attack vectors.
Another element to consider is the need for service-based pentesting. Specifically, compliance standards like SOC2, ISO, HITRUST, etc. require that a company hire a security service to do pen testing — having a product do pentesting is insufficient.
Therefore, the market has two extremes. One side of the market has companies doing pentesting just to fulfill compliance. These companies need to use a security service, so they likely won’t want to buy a product like this. The other side wants to detect sophisticated attacks. This likely requires a high-quality service or an in-house team that maintains context and can interact with the engineering and product teams. This type of customer also would probably use tooling to augment and do automated pentesting, but that’s mainly to assure good posture and catch any deviations. So, these tools serve as supplements rather than replacements, so it’s likely hard to get large budgets for this.
As a result, the markets for these types of tools are small. The side of the market with the most demand is using this tool as a part of its strategy rather than the core. If forced to reduce its budget, it will likely spend more money on high-quality services than these tools because it’s unlikely these tools will find these sophisticated attacks.
I talked about how most security tools are too abstract in their goals. This definitely feels like one of them because it deals with theoretical attacks, so it’s hard to know the ROI because it’s unclear if it’s prevented real attacks. With that said, this feels like it adds more complexity to a company’s security operation at a time when efficiency is key.
How do these companies move forward?
It’s clear that this is a “nice to have” tool rather than a “must have.” In a noisy security environment with decreasing budgets and/or more efficiency demands, they have to show clear ROI. Some companies like Praetorian are supplementing the product with services. Others are positioning themselves as automated red teaming or attack surface management.
The key is that these products need to demonstrate specific, measurable business outcomes to justify investment. One way is to position itself as reducing the remediation costs of pentests by detecting issues earlier, leading to fewer findings during the service-based pentest. This seems measurable because the product could find problems that would otherwise appear in a pentest and lead to unexpected work. This feels like a good direction to go in, but it’s not sticky and has a ceiling on its value. This feels like the posture management portion of Wiz, which seems easy for people to eventually build themselves.
Another way is to integrate into SIEMs and other incident detection tools. This can give an advantage by allowing these tools to learn what attacks might occur based on previous incidents. This way, it can try to run attacks to see if certain security events might escalate into incidents easily based on certain setups. Similarly, it can constantly test once you’ve made fixes after a security incident to ensure the same incident doesn’t happen again — you’ve properly remediated the issue. Nothing is worse than having the same security incident happen twice for the same root cause. It also can test for whether similar exploits exist even after a patch.
Similar to what EDR has done with MDR, it can provide a managed service layer on top. There will be some human interaction to try to detect more sophisticated attacks. The main limitation of current tools is that it doesn’t detect the advanced exploits that a good pentesting service can detect. As a result, having a managed service layer can close the gap there and lead to lower operational costs, and it’s possible to detect more sophisticated attacks. That will likely allow it to capture some part of the market, especially in companies that are trying to reduce security service costs.
These tools are currently looking to find a place in the company’s security stack. It seems that only mature companies need this. However, it’s important for these tools to start focusing on ways to demonstrate business value by proving they are preventing future incidents or reducing operational/service costs for the company. What seems to be happening is that these products are still trying to figure out how much automation vs. services (humans) to use in their product. It’s going to be a tricky balance. Right now, the value ceiling is low, and it’ll be interesting to see how these products try to capture more value.