Bug bounties misguided attempt to crowdsource security, report suggests

News by Tom Reeve

Attempts by companies to harness the wisdom of the crowd, or cut corners on security research depending on your perspective, by launching bug bounty programmes are misguided, researchers say.

Big payouts for bug bounties are being given only to a few elite security researchers, leaving the vast majority scrambling for the leftovers, according to an analysis of the market.

Security company Trail of Bits based its conclusion on ‘Fixing a hole: the labour market for bugs’, a chapter in a new book from MIT Press called New Solutions for Cybersecurity.

It calls into question the whole rationale for setting up bug bounty programmes, the company said in a blog post.

Bug bounty programmes are being launched by a host of companies looking to crowd source their security, such as Hyatt Hotels and even the European Union.

Data from the article shows that big payouts go consistently to an elite few bug hunters, while the vast majority of those who sign up to bug bounty sites like HackerOne rarely if ever find anything worth a payout.

And those elite who are getting the big payouts are not earning nearly as much as they can earn as top-level security staff, leading the company to suggest that most bug hunters do it as a sideline.

The idea that by signing up to a bug bounty programme your company is suddenly going to have hundreds of thousands of eyes scrutinising your code – and that you will only have to payout in the event that they find something – is misguided, it said.

"If you're a company, you don't want your security dependent on whether or not a single high-performing researcher is dedicating enough hobby time to look at your product," the company said.

In addition, open bug bounties attract a "tsunami of trivial, non-issue, and duplicate bugs", the authors of Fixing a Hole said, quoting statistics from Google, Facebook and GitHub that showed that only four to five percent of reported bugs were eligible for payment.

While the authors of Fixing a Hole suggest that bug bounties should be made invitation-only to incentivise the elite, but Trail of Bits argued that "hiring security consultants under terms and conditions that can be controlled seems more practical".

An analysis of data from HackerOne and Facebook’s bug bounty programme by Trail of Bits found that seven percent of participants with 10 or more bug reports each were paid 1,622 bounties while the remainder of the group, 93 percent, were paid for 2,523.

Further analysis of the top performers showed there earnings were not very impressive. For the Facebook group, the top hunters earned US$ 34,255 (£26,679) a year for an average of 0.87 bugs per month while the HackerOne elite bug hunters averaged 1.17 bugs per month and earned US$ 16,544 (£12,886) in a year. Skewing these averages of course are the occasional, very big payouts, such as the US$ 60,000 (£46,733) paid by Google for a Chromium bug.

The authors of Fixing a Hole make three recommendations:

  1. Keep the talent pool exclusive through invite-only programmes that are closed to the public. This ensures that the most talented will not lose any bounties to lesser talent—even the low-hanging fruit.

  2. Escalate prices with successive valid submissions to deter people from straying to other programmes.

  3. Offer grants to talented researchers, and pay them even if no bugs are found.

However, Trail of Bits suggests – not surprisingly given its service offering – that this is not much different from hiring a consulting firm to do a code audit.

Meanwhile, for those bug hunters whose hats are not white, shading into grey or possibly all black, there is the zero-day market to consider. While the economics of this market are less well know, various companies promise payouts in the millions of dollars for the best vulnerabilities.

Find this article useful?

Get more great articles like this in your inbox every lunchtime

Video and interviews