How the AI security space will evolve
LLMs are the reason that most AI security companies won't last
Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer or any other entities with which I am affiliated.
We’re hiring for several roles at Headway in our Trust organization, led by the fearless Susan Chiang. Specifically, we’re hiring a software engineering manager, a product manager, and a product security engineer.
This post is coming a bit late, and it might be considered a 1.5 post. I was at our company hackathon last week, where I finally had some free time to try out many cutting-edge AI tools, such as Anthropic. I played around and saw the AI developer tools, such as Cursor and Copilot in action. I have to say it’s a different feeling being hands than seeing the demo. This is what I wanted to write about. Initially, I was confused by the range of AI security startups, and then I realized that it’s possible many security people haven’t tried a lot of these tools directly and actually developed in or with them. The lack of hands-on experience might be driving a lot of the noise in AI security.
As many of you know, I’ve been writing about the intersection of security and AI.
It goes without saying that there’s a lot of activity, and it’s frankly hard to keep up. I don’t blame anyone, especially in security. There’s already enough security activity, and now there’s AI activity on top of it.
I wrote that a lot is unknown about AI security, but in this newsletter, I want to revise that statement — a lot of the future market is unknown. That seems like a somewhat obvious statement. Of course, we don’t know what the future markets will be like! If we did, then we would not need experts to predict the market or spend time betting on the market. However, what I mean is that the AI security market is more unpredictable compared to other security markets. We don’t know how AI will mature and how companies will adopt maturing AI tools. In fact, if DeepSeek’s achievement proved anything, it’s that we can’t predict AI’s innovation progress. No one guessed that we could come up with AI models so cheaply so quickly. This likely means that AI models will improve faster than we imagined.
Now, that I’ve probably stated some obvious facts. What does this mean for AI security? (other than the fact, that it makes building a startup harder.)
What is the state of AI usage?
There are different types of AI models, and they are used differently. Large language models (LLMs), so the likes of Anthropic, DeepSeek, and OpenAI are prompt-based. This means you enter a prompt to give the LLM “instructions” and a corresponding message, and then it outputs a result. It’s pretty straightforward to use, and most people are using this. You don’t need very little, if any, machine learning or AI background to do this — you just send the prompt and message to Anthropic or OpenAI and wait for a response based on the “instructions” in the prompt.
There’s the more “involved” AI, which requires you to build your own models. This requires training data to build a model, and after that, you can put in your inputs and get outputs. You have to build a model for each application. This will require you to have some machine learning and AI knowledge. The impressive part about LLMs is that these models are already built and are general enough to be used for a wide range of applications without any need for specialized training!
What does this mean? Since LLMs are substantially easier to use, most companies and developers will use LLMs rather than going out to build and train their own models. Fly.io’s blog post confirms this trend.
But, of course, what does this mean for security?
Most companies looking to monitor models will face a small market or an uncertain market. It’s not clear when companies will start to use their own models (if ever!). If we look at the set of Latio Tech’s AI security startups. It seems like the AI model-focused ones like Noma, Operant, Aim security are unlikely to gain any sustainable traction.
However, LLM-focused solutions like Lasso and Prompt Security might get more initial traction because they are focused on tracking LLM-specific issues, such as prompts and messages.
My broader take is that at this time, it doesn’t make sense to have an LLM-security-focused company. Most developers using LLMs are already logging and versioning prompts, messages, and outputs for quality control. It might be useful to have a tool to scan these logs and detect anomalous messages and outputs. However, it seems people want to do this manually or with basic rules, i.e. build it in-house without AI, because it doesn’t make sense to have another LLM monitor the LLM.
Another interesting fact is that it’s easy to get about 80 percent of the result you want in a short period of time, but it takes 4-5 times longer to make another 10-15 percent improvement. There’s a LinkedIn blog that said that their team produced 80 percent of the result in a month, but it took another four months to achieve 95 percent. Of course, there’s some luck involved, and results might vary. Either way, this does give a good sense of how LLM usage for tasks works. As a result, developers need to constantly log the messages and outputs to improve the prompts.
Overall, this period is likely still less time than developing an in-house model, and it doesn’t require AI/ML specialists! Especially as these LLMs improve, most companies will find that LLMs are sufficient for their AI/ML needs. In many ways, this reflects similarly to the cloud where most companies likely don’t need to have their own cloud. That is, public clouds are sufficient until they reach a certain scale despite what Martin Casado says in his article about the trillion-dollar cloud paradox.
A slight digression here. Will this make it so that people will move back to public clouds? It makes sense to have your own cloud to work on “predictable” and inelastic workloads like AI, but will AI clouds make these workloads obsolete and thus private clouds less attractive?
Anyway, back to the point. The largest market will be ones that can “monitor” LLM usage whatever that means. For now, it seems that it’s more of a developer tool to help with prompt engineering and improving their usage of LLMs. This could be similar to a Snyk-like situation where they can sell useful dashboards and reports to security, but I don’t see a lot of value in a dedicated security tool.
What about security for companies building their own models?
Companies will try to build their own models, but the number of companies doing this will be small unless having their own models is strategic. Although small, it will be an important market. There’s still some room for an AI security company here, but the tricky part of this market is that companies have yet to figure out what to do with their own models vs. LLMs, and it could affect the type of AI security tool and the overall market size. Another side here is whether a company monitoring infrastructure workloads like Wiz or an MDR will take this market. I do think it’s likely, but I also think it’s possible that in the future AI workloads will be as large as application/operational workloads in the way that analytics workloads are as large (if not larger in some cases) than operational workloads, which led to the rise of data warehouses like Snowflake and Databricks. As a result, there will be a need for a new category of infrastructure security company for AI workloads.
Similar to the cloud, it’s not clear when the increase in in-house AI models will occur. It feels that the better LLMs get, the less likely companies will feel the need to have their own AI models. At least right now, it feels that the speed of LLM improvement will outpace the ease of creating your own AI models at least in the next 5-10 years.
Security for other AI tools
Other than LLMs, other tools might use either LLMs or their own AI models. There are many of these, but some notable ones are developer-related, such as Cursor and Copilot. There are also applications, such as Glean, that will help ingest large amounts of information, and I do think at some point, we will see an increase in autonomous agents and operators.
How will security handle these situations? Since these agents will take action without much or any human guidance, these actions need to be trackable and auditable. If they are deployed within your own infrastructure, then this is easy, but it’s hard when they involve other SaaS tools that you don’t control. I believe that security teams will demand that these agents provide an audit trail, and it’s likely that MDRs or SIEMs will analyze. It’s possible that there will be tools to handle agent audit trails if they are different enough, but right now, it’s not clear that they are.
So, what do I believe will happen?
I’m becoming more bearish on AI security companies. There might be some room for a couple of security companies that focus on monitoring in-house AI models, but I believe that companies will move away from these as LLMs improve. LLM security companies will either act as debugging tools for developers working with LLMs, or the debugging tools for LLMs will make these LLM security companies obsolete. There’s no reason to have both. Overall, the value seems a bit low here because it feels like just a logging feature.
There might be some tools to add visibility for other AI tools people might using, but I imagine that these tools will provide audit logs that SIEMs and/or MDRs will ingest. As long as LLMs continue to get better, the market for AI security companies seems to be shrinking.