Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer or any other entities with which I am affiliated.
I’ve gone to some security dinners recently, and one topic that almost always comes up is the concept of security reviews. Although I don’t suggest we completely get rid of them, I do believe we need to have a reset. In other words, we need to have security reviews or processes that make sense for our current development practices.
What are security reviews?
Security reviews come in all shapes and sizes. However, in most companies, the security team reviews design documents and joins architecture meetings to ensure certain security requirements are met. The requirements come in different forms, and the details vary. Most of the time, security design reviews are only required for large changes. They rarely are required for smaller features. Otherwise, it would slow down development too much.
What are security teams looking for? It’s very similar to a legal review of a contract, where the design is the contract. Security teams rarely have the power to block a rollout unless it’s extremely egregious. They just discuss the risks, and the development team decides what they want to take on.
There are usually checklists to focus on the basics, such as understanding data flows, access, potential vulnerabilities, etc. There are also specific security risks the development team has uncovered. They will discuss the tradeoffs and mitigations with the security team, and then the security team will tell them what should be prioritized, and let the development team ultimately decide. It’s tough for the security team to provide a complete assessment because they lack context. I’ll discuss this more below.
Why are security reviews problematic?
In short, I believe doing security reviews is similar to “seagulling,” which happens regularly in management. The problems stem from the fact that security teams rarely have the context or power to effect any sort of meaningful change. At best, the whole security review process feels like a formality.
Security reviews don’t make sense in the cloud and agile world. Development cycles are fast, and it’s rare to have such major changes. Even with major changes, it’s hard to do a security review because the project is complex. As a result, security has to be involved from the beginning, but security rarely has the skillset to meaningfully contribute to those types of designs. If security had the skillset, they should offer to embed or assist the team with the development itself.
Ultimately, security lacks real power to effect change. In many companies, security reviews are part of the deployment process, and they end up feeling like a formality. More specifically, people are just looking for security “signoff” rather than any sort of meaningful feedback. Security is caught in a tough position. They can’t block a project because others will find ways around this in the future by breaking up the project into features or workflows that don’t require security review. They also enter so late into the design process that they rarely have the context to properly assess the design. They can present risks because product launches and business needs always trump those of security. It feels that security reviews are set up for failure and as a way to place blame on security when something goes wrong. That is, if security believed it was such a major issue, they should have blocked it.
Especially on fast-moving or ambitious projects, the design regularly changes. As the project develops, usually there are roadblocks or tradeoffs that need to be. Anyone who’s done development knows these are hard to anticipate. It’s too disruptive to consult security on all these changes, which means that security doesn’t have a chance to validate that the risks they signed off on are actually the risks that end up in the finished product.
It seems that security reviews feel dated. It might have made sense in the waterfall world where development and release cycles happened once or twice a year. However, it doesn’t scale well with current practices. In fact, they seem operationally heavy, and it’s not clear how much value they bring to an organization and how much risk they reduce. Of course, one can point to anecdotes where they were helpful or found an issue, but if a company looks at the amount of time spent on security reviews for both development and security teams, it seems that the resources are probably better spent developing more security guardrails. The risk-benefit of security reviews in their current state doesn’t make sense, especially for smaller companies, which have more limited resources.
What are some alternatives?
We need a security review process that’s efficient and/or effective. Sometimes, it’s hard to do both, but currently, it seems that process is neither. Here are a few ideas. None of these are meant to be mutually exclusive.
First, we should have a security checklist as part of a service standard. Most companies have some sort of service standard that provides a paved road to release a feature or service. These standards give the development team the green light to deploy a service in a lightweight manner without having to consult numerous teams like DevOps, other product teams, etc. There should be a security portion, and if teams meet these basic security checks, they are good to go. If they have questions or want an exception, they can reach out to the security team. Hopefully, they can design a checklist where exceptions are limited. Policies stop being meaningful if there are more exceptions than rules!
For more security-sensitive products, there can be a “sister” security engineering team or embedded security engineers. These are software engineers who work with the core team to ensure the right security features are developed and validate that they work. They will also work on the product, so they have the proper context when parts of the design change. This is common in regulated and sensitive industries like healthcare and finance. This way, the security can see the product from start to finish as well as contribute and have context. However, this does require finding a software engineer that understands security.
I’ve just described two sides of the spectrum. For situations in the middle, you can embed a security engineer part-time to work with the team, or many companies have a security champions program. This works well if there are security-minded engineers at the company or on the team in question.
The idea here is that we have to have a way to scale security thinking throughout the organization. However, on security-critical projects, it’s actually more efficient to have a security engineering team work alongside the team. It helps provide context while making progress on both security and the product itself rather than having constant interactions with security to share context and get feedback. Currently, that becomes tedious that many times, these interactions between the engineering team and security don’t happen at all!
Takeaway
The current security review process is not working. It doesn’t make sense for our current development practices. We need to find ways to better scale security efforts rather than imposing security reviews that have become more of a “checkbox” than actually solving problems. This likely might require involving software engineers and security being more involved in engineering processes, which requires more software engineering skillsets on the security team.