Frankly Speaking - Analyzing the Reddit Hack

Preventing phishing attacks is extraordinarily difficult

Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer or any other entities with which I am affiliated.

I’ve gotten many requests to collaborate and/or do a guest post. Please keep them coming! I’m excited to collaborate with many of you, especially those that have Substack accounts!

LET’S BE FRANK

I’ve written a series of posts analyzing various high-profile hacks, such as the LastPass hack and the CircleCI hack. This past week, I became aware of the Reddit hack, which they announced in a Reddit post. In this newsletter, I will analyze the announcement and provide some of my thoughts.

In short (or tl;dr like most folks on Reddit like to say), I thought this was a good disclosure and provided clarity to those affected.

Let’s go through the announcement paragraph by paragraph.

TL:DR Based on our investigation so far, Reddit user passwords and accounts are safe, but on Sunday night (pacific time), Reddit systems were hacked as a result of a sophisticated and highly-targeted phishing attack. They gained access to some internal documents, code, and some internal business systems.

This was a great summary. It was clear that core user assets were secure and succinctly explained what happened and the resulting impact. It gave users clarity and said that no action needed to be taken from the start. I think it would have benefitted to tell users that no action needed to be taken on their part or suggest that they add 2FA to their accounts and rotate passwords as a precaution. Other than that, good job, Reddit security and communication team!

What Happened?

On late (PST) February 5, 2023, we became aware of a sophisticated phishing campaign that targeted Reddit employees. As in most phishing campaigns, the attacker sent out plausible-sounding prompts pointing employees to a website that cloned the behavior of our intranet gateway, in an attempt to steal credentials and second-factor tokens.

I’m not going to re-post this whole section, but this was a clear explanation that showed there was a targeted, sophisticated phishing campaign. It seems that the attackers knew the intricacies of the internal system enough to launch such an attack and social engineer an employee. These attacks are typically hard to defend against since they are low effort but high impact. They also only require one set of credentials, and it seems that they were also able to trick the employee into giving up his/her 2FA token via phishing too.

After successfully obtaining a single employee’s credentials, the attacker gained access to some internal docs, code, as well as some internal dashboards and business systems. We show no indications of breach of our primary production systems (the parts of our stack that run Reddit and store the majority of our data).

However, it does show that they locked down their production systems very well. I could imagine that they require all employees with access to production systems to use a 2FA method like WebAuthn or FIDO, which is phishing resistant.

Our goal is to fully understand and prevent future incidents of this nature, and we will use this post to provide any additional updates as we learn and can share more. So far, it also appears that many of the lessons we learned five years ago have continued to be useful.

It seems like they took the right mitigation steps and learned from their hack five years ago. They no longer use SMS-based 2FA and have upgraded to token-based 2FA. They also seem to have good education around phishing given the employee self-reported, thus reducing potential damage. Moreover, Reddit has managed to isolate the blast radius, i.e. access to the internal system didn’t allow someone to escalate access to the production systems.

They went on to provide some basic guidance to users and an AMA, which seems useful to answer most of the questions people had.

Takeaway

Overall, this short announcement covered a lot of important information and provided clarity to users. It was well-organized and left users feeling that their information was safe and that Reddit took all the appropriate steps.

What are my thoughts on this security incident? Well, phishing is hard to prevent, which is annoying. Everyone has to use email, and it just takes one successful phishing attempt to steal credentials. There are a few ways to “solve” this, and it seems that Reddit already implemented many of them.

First, try to reduce phishing. One way to do this is to focus on phishing education and convince people to be alerted. Although everyone dislikes phishing training, the data is clear that it does reduce the number of successful phishing attempts. Moreover, an organization can identify high-risk individuals, but it should be clear that falling for phishing attempts is perfectly normal. This will encourage rather than discourage employees to report phishing incidents, which seemed like what happened here.

Second, reduce potential avenues for phishing. Authenticated services like Okta and Slack has reduced attempts at phishing. Encourage employees to communicate primarily through Slack and share information through there and use secure channels, like 1Password to share sensitive information like keys. Also, if an employee receives an email about going into a system, they should go through Okta rather than click on the link. This is hard and just requires building muscle in the organization. However, encouraging everyone to do it sets an important example. There is no other technical solution to this problem.

Third, assume phishing will happen and reduce the blast radius. It seems like this has happened at Reddit. Getting into the internal system didn’t grant them access to the production system. It’s important to consider having elevated security, such as WebAuthN, for employees who need to access production systems. Those systems should be heavily monitored, and there should be some friction for access. Following the principle of least privilege is extraordinarily important here.

Finally, logging is important! There should be access logs for access to all important systems, so even if someone were phished, it’s possible to trace the damage. Similarly, there should be email logs to see who else might have clicked a phishing email that slipped through the filters.

Well… more finally, it’s important to practice having an incident so that there is organization and clear communication standards. It’s best to establish some ground rules ahead of time and agree on important decision-making criteria. This is not something you want to discuss under duress! There needs to be clear ownership of the various parts of an incident. Otherwise, you will waste time and ultimately make a poor decision.

I applaud the Reddit security team for doing a great job here from what I can tell! Security incidents are stressful, but they seemed to have had preventative measures in place and have an organized way of handling a potential incident.