Frankly Speaking - We need to fix bug bounty

It's important but broken.

Disclaimer: Opinions expressed are solely my own and do not express the views or opinions of my employer or any other entities with which I am affiliated.

Happy Thanksgiving everyone! After a hectic year, I hope everyone has some downtime to spend with their families. Again, I appreciate all the support. If any of you have some leftover professional development budget, please consider spending it on a paid subscription! It helps keep this blog going.

LET’S BE FRANK

Recently, there’s been a lot of talk about bug bounty, especially given the verdict around the Joe Sullivan and Uber case. In my opinion, bug bounty is one of those programs where the tail end of issues defines the value, but on a regular basis, it is just a pain to manage. Honestly, I don’t think I’ve spoken to one security professional that likes bug bounty, but everyone believes they should suck it up and do it.

Why is this? Why does managing bug bounty have to be so painful? I’ll talk about why the current state of bug bounty is so dire, and why we need to fix it now. I will also discuss how companies should think about bug bounty from a business objectives perspective.

The rise of the bug bounty

Honestly, bug bounties are nothing new. I’m not going to insult any readers by describing what a bug bounty is, but recently, there’s been a rise in formalized platforms. Before these platforms, many sites had informal security vulnerability disclosure programs (and still do). White hat hackers would submit security issues and way to reproduce them. Many times, people would do it to help improve an application or website they enjoyed. There was no expectation of payment. Many researchers actually did it in their free time and thought it was an important duty as someone in the security community.

Times have really changed. Bug bounty platforms emerged along with Tor and the dark web where black hat hackers can now sell vulnerabilities. Finding security vulnerabilities became a business! Bug bounty platforms were a consequence of this. They allowed a white hat hacker or security researcher to find bugs in various websites rather than searching around for different applications. There was guaranteed payment, so many security researchers now make a living finding vulnerabilities.

It seemed like promising, and it allowed companies to crowdsource finding bugs. It was a more formal way for others to find vulnerabilities in a controlled environment and not on the live site. Companies could better communicate and work with researchers. It definitely seemed like these platforms were an improvement.

However, a new host of problems emerged and now plague this security tool.

The problems with bug bounty

I can probably write multiple blog posts just about the various issues with bug bounty, so I won’t be able to highlight all of them here. However, I want to discuss a few key ones.

The main issue is that before bug bounty platforms, finding vulnerabilities was mostly a side project for the security team unless you were a high trafficked site. Now, with these platforms, it’s better for the researchers but worse for a company because there are more people trying to find vulnerabilities on the site. As a result, most companies have to develop a program and commit operational power to this, which can be extremely costly and time consuming. Many factors contribute to this.

Many security researchers are lazy.
Let’s be honest. Most security researchers aren’t providing a white glove service and looking diligently for bugs. They are playing the volume game and hoping a bug sticks. As a result, most bug reports aren’t well-written. The worst of it is that most researchers just run automated tools and scanning without actually spending time to learn the application to find real issues. This leads to a highly noisy bug reports, and most discovered problems aren’t worth fixing or even security issues at all.

Environments are unrealistic.
Before bug bounty platforms, security researchers used to find vulnerabilities on the main version of the application. This is problematic for many reasons. Many times, they caused disruptions in actual sites or left evidence that allowed for black hat hackers to do damage. However, the bug bounty platforms have allowed for more controlled environments by allowing companies to set up separate infrastructure for researchers. This is beneficial and problematic at the same time. We don’t run into the problems described above, but most controlled environments don’t replicate the actual production one. This results in finding bugs that don’t exist in production or missing bugs that actually exist in production.

Reproduction is hard and time consuming.
The most time consuming part of bug bounty is to reproduce the bug. Although it helps that the researchers create reproduction steps, independent validation still takes time. The issue is that most security teams rely on engineering teams to do the reproduction, so it is an inefficient process that creates a huge burden on the engineering team. This is another reason to have more software engineers on the security team.

Triaging is highly inefficient.
This is related to the reproduction issue. The hardest part is to figure out if a bug is actually a bug that 1. hasn’t been discovered and 2. is actually a bug. Most security teams with software engineers can solve 2, but 1 is tricky. It’s possible that the company already knows about the bug but just hasn’t had time to fix it. Relatedly, it’s possible another researcher found the bug earlier. However, there’s no way to give this information a priori to the researcher, so researchers waste a lot of time on duplicated bugs. Bug bounty platforms try to find ways to do this. There is really no way to do this properly without releasing sensitive information to the researcher.

Is bug bounty worth it?

This is the ultimate question. There’s a ton of anecdotal evidence that supports the bug bounty program, but very few leaders have spent the time to figure out the quantitative and qualitative costs of bug bounty and frame it in terms of business objectives. More specifically, is bug bounty a good use of resources, and are those resources actually helping the company become more secure? Can these resources be dedicated somewhere else that could have prevented these bugs? Those are hard to say and depend on the company, but I would argue the answer is not as obvious as many think.

Bug bounties do feel “lightweight,” but they create a much larger tax on an organization, especially a resource constrained one, than many think. However, creating strong processes and programs can help mitigate some of this. Unfortunately, the reason that we don’t see more of this type of thinking is that those who run and create these programs don’t have the right business context. They are mostly old school security leaders who are focused on threat risk reduction in isolation rather than thinking about it in terms of business risk reduction. Bug bounties do seem pretty harmless and sound operationally lightweight, and the tail end of discovered issues create most of the hype. However, it’s like venture capital. Sometimes, the winners create outsized value, but for most funds, this is not the case.

So what am I saying here? I think we need to fix aspects of bug bounty, but before we start thinking about it, we need to truly understand quantify the impacts on business. This way, we can make sure to fix the right problems and think of ways to develop programs that make sense.

Of course, there’s different levels of effort a company can put into bug bounty, but in order to find the right balance, the right people with the right context have to be in the room to evaluate the risk-reward of the program. This will always be the ongoing challenge.