Who's responsible in the Snowflake breaches?
It's tricky to split a shared responsibility
Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer or any other entities with which I am affiliated.
Over the past week, multiple large companies, such as Ticketmaster, Satander, and Advanced Auto Parts, reported data breaches linked to their Snowflake accounts. However, as far as we can tell, Snowflake itself wasn’t directly breached. Initial investigations show that hackers used malware to obtain the Snowflake credentials of the company's employees. Sure, the malware likely obtained other credentials, but why is Snowflake getting all the attention?
One major reason is that Snowflake itself is a “high-value” target because many customers store sensitive data, so obtaining credentials to that has a higher blast radius compared to another SaaS application like Asana or Datadog.
Another reason is that it seems that Snowflake allowed customers the option not to enforce multi-factor authentication (MFA) by default. Usually, obtaining credentials alone isn’t enough to access a high-value corporate application because integration with Okta or another SSO provider enforces MFA, or the application itself enforces MFA, especially since Snowflake is a highly sensitive application. This brings up the question: why didn’t Snowflake force MFA, and do they really need to?
How does Snowflake MFA work?
Before we answer the question above, it’s worth diving into the details of how Snowflake sets up its authentication and the product security experience. The reason is that every application has a different way of setting up its authentication. This is an amazingly frustrating experience and a huge pain point that SaaS needs to address. It creates huge operational burdens on both IT and security because they have to read the documentation for every single SaaS application to figure out how to set up authentication securely. It’s interesting because we have standards for authentication, such as SAML and FIDO2, etc., which govern how authentication should happen securely, but we don’t have standards for authentication setup. If you set up authentication for three SaaS applications, they will follow three different sets of instructions. Talking about this frustration further is likely the topic of another newsletter.
In Snowflake, there are two main ways to set up MFA. The users themselves set up MFA, or you can integrate with a SAML/SSO provider like Okta that enforces MFA. This seems pretty straightforward, but implementing MFA is never the hard part. The hard part is enforcing MFA.
One way to do this is to force everyone to log in through SSO. Setting up SSO is somewhat challenging in Snowflake. You have to run an SQL query. Although the documentation tells you how to do it, it might be daunting for a security person who might have limited engineering experience. On top of that, it’s a whole other set of queries to convert existing users over to ensure, in the eyes of Snowflake, they were provisioned via the SAML provider. Let’s not even get started about SCIM. At this point, if users had passwords before, they can still log in, so then it’s another query to remove passwords so that you force them to log in via SSO rather than password. You also have to set up a service account as a “break glass” mechanism in case SSO is down, or you mess up setting up SSO. This is complicated for IT or security to figure out, and it likely requires help from the team owning Snowflake, i.e. the data team. If that’s the case, there will likely be friction, and as a result, it’ll get deprioritized or not fully completed. It also requires a fair amount of maintenance. Complicated product security experiences make it hard for security to do its job, but more on that later.
Another way is to force MFA onto users. However, Snowflake doesn’t automatically enroll or require MFA for its users. It just strongly recommends it, so it’s up to the data team and security team to socialize this. However, MFA for Snowflake is hard to set up and requires downloading Duo, which is itself atypical. This also makes it tough for security and operationally intensive because they have to regularly audit to ensure users have MFA enabled. Of course, security can work with data to do this, but that’s another variable to deal with. This is a case where security and data need to share context, which just adds friction, increasing the probability that something slips through the cracks. This is what seems to have happened with the breached companies.
A complicated product security experience
Overall, it feels like the product experience here of enforcing MFA is not ideal. Snowflake is a great product, but it’s not surprise because I have always held the belief that cloud providers or more generally usage-based SaaS companies don’t care about security.
Two main reasons apply to this situation. First, since Snowflake is a usage-based product, it’s incentivized to increase customer usage. Having complicated logins goes against that goal. Second, the end users themselves, i.e. data, typically don’t have enough context about security, so it’s hard for them to appreciate the security features. As a result, from a company investment standpoint, it’s unclear how much to invest in product security.
This brings up an interesting question: why don’t they automatically enforce MFA for users in response to this, or require it by default? The answer is that it would break data workflows.
There’s a key nuance. Snowflake has many integrations, but they aren’t all native. This means that in order to access Snowflake data, customers have to create a user for this integration. Some integrations support keypairs, which makes it easy to rotate and distinguish from human users. However, many don’t, so it requires a password. It’s hard to do MFA for machine-to-machine interactions. Enforcing MFA by default would increase friction and/or require the integration to make changes, and since this type of user is hard to differentiate from an actual user, turning on MFA for all users to protect accounts is likely complex. Again, this counts as a complicated product security experience around authentication and authorization that pushes much of the effort to the end user.
Who’s responsible? Snowflake or the customer?
This is the question, and it’s not clear in my mind. It’s not obviously Snowflake’s responsibility because it’s not a direct breach of its systems, unlike the United Healthcare breach. Snowflake could argue that customers should have had stronger endpoint protections to prevent malware and enforced MFA. The customers who followed Snowflake’s best practices most likely were unaffected. The other side would say that Snowflake should have made better product decisions by forcing MFA by default.
Either way, I do feel for the Snowflake security team. They have an incredibly hard job right now because they have to respond to this incident, dealing with the consequences of others’ decisions.
I do see Snowflake’s side. For example, suppose a user uses a weak password and turns off MFA for their email. If they are hacked, who is responsible? It would be a huge burden for email providers to take full responsibility for that. However, current email providers do make it easy to enforce stronger passwords and MFA. They also have mechanisms to detect malicious activity and figure out ways to ask for another factor or lock you out of your account. That’s because email is a high-value target, so email providers have invested substantially in security.
The main difference is that email is a competitive space. If someone doesn’t believe his/her email provider is doing well in security, he/she can move to another provider. Security is a differentiator. Snowflake hasn’t faced such strong competitive forces. The alternatives aren’t great. I’m not saying this is a good excuse, but rather, the market hasn’t forced them to invest in security. I get it. It’s generally hard to convince executives to spend on something defensive, where they can just pass that “cost” and “responsibility” onto the customer.
Snowflake’s mindset is outdated. It was more acceptable for on-premise software where the customer had more control over the infrastructure and network. They can more easily have better login controls and visibility. However, with SaaS, this is more difficult because the only real visibility for the customer is the audit log provided by the software. A big part of SaaS is that the software provider is now managing the infrastructure, so they bear more responsibility for both monitoring malicious activity and ensuring more seamless product security experiences. I’m not saying that Snowflake should bear full responsibility, but it should bear more responsibility than it initially did.
The world isn’t black and white here. I agree that Snowflake isn’t fully to blame, and in fact, they shouldn’t take the full blame because, at the end of the day, it’s a configuration issue. However, compared to other software, Snowflake does have a worse product security experience — they make it hard (but not impossible) to do security properly.
There is a learning. It seems that the executive team didn’t properly consider this risk, and there wasn’t proper security involvement in the product. How did that break down? I don’t know and can only speculate. Should Snowflake have done more? Probably, but how do we make companies like Snowflake more responsible for product security and make sure security incentives are more aligned? I don’t know. I think it’s up to the market and industry to decide. In the worst case, we might need to resort to regulatory forces. But, this should be a wakeup call to understand what we are dealing with in the world of SaaS.