Ilia Kolochenko, CEO of High-Tech Bridge
Ilia Kolochenko, CEO of High-Tech Bridge

(Note: This article intentionally concentrates on web applications and omits Bug Bounties for stand-alone software (eg Chrome) as they have a different approach and rules.)

Are Bug Bounties suitable to test your corporate web applications without putting them and your infrastructure at risk?

Bounties' “pay-per-result” remuneration may look attractive for many CISOs who have to optimise their budgets to cover newly-appearing risks and threats. Crowd security-testing also makes a lot of sense from a technical point of view - the more researchers with different skills and experience participate, the more vulnerabilities and weaknesses they may find. However, Bug Bounties also have some serious pitfalls:

1. Mismatch with your risk management strategy and risk appetite

A vulnerability you may be perfectly aware of, and that you do not plan to remediate immediately for a reason (eg compatibility problem or agreement with supplier), may be quickly reported by several researchers claiming the bounty. A refusal to pay will certainly provoke researchers' rage, who will stop reporting vulnerabilities in the best-case scenario. Yes, you can specify any conditions and criteria for vulnerability submissions, however few researchers will fully read them and even fewer will practically respect them. The recent Facebook bounty scandal is a good example when company and researcher may have different visions on the scope of testing and related risks.

2. Solid technical, human and financial resources requirement

Some people tend to think that a Bug Bounty can seriously reduce vulnerability testing costs as you pay only for results. For the majority of cases this assumption is wrong, as a poorly-implemented Bug Bounty will just spoil your relations with the infosec community and damage your corporate reputation. Some researchers will even intentionally proceed to Full Disclosure in revenge for lack of responsiveness. If you launch a Bug Bounty, make sure you have enough technical and human resources to handle, analyse and properly follow-up the avalanche of submissions - every single submission should be properly answered in a timely manner. Oracle, which does not have a bounty programme, attracted negative media attention last year, when its CSO complained about tons of garbage vulnerability submissions, including copy-pasted reports from automated vulnerability scanners.

3. Black Hats may easily hide amongst legitimate researchers

Once a Bug Bounty is announced, many legitimate researchers from all over the world will start probing your systems. Quite often they may go beyond the initially defined scope of testing, trying to compromise a secondary system to get access to the target (eg to get source codes of your web application stored on SVN server). Newbies will just launch numerous vulnerability scanners against your networks. Such an environment is ideal for Black Hats to hide in the traffic noise, and a nightmare for your SIEM. Some researchers may also use unexpected and dangerous testing methodologies – up to intentionally running DoS attacks against your infrastructure.

4. Costs can get out of control much quicker than you think

At first glance, Bug Bounties offer a predefined cost you may easily keep under control. In some cases this is true. In others however you will have to deal with numerous ‘race conditions' when the same vulnerability is reported by several researchers at once. Refusing bounty payment will provoke many negative emotions. You can restrict awardable vulnerability types (eg to exclude low-risk flaws), but you may be suddenly surprised by how many rewardable flaws are finally reported, boosting your costs.

Several Bug Bounty management companies recently entered the market, offering a sort of intermediary service between your Bug Bounty and researchers, to help mitigate these challenges. Although they do a good job, helping both businesses and the researchers, even managed Bug Bounties have significant limits in comparison to traditional web security:

1. Bug Bounty will not replace continuous monitoring, SDLC or WAF

Today, continuous monitoring is a vital part of almost any information security strategy. But, since you can't force researchers to respect any sort of SLA – they test whenever and however they want - you certainly shouldn't expect a Bug Bounty to compete with the reliability of a 24/7 monitoring system. Similar to classic penetration testing or vulnerability scanning, Bug Bounties aim to detect vulnerabilities, not patch them or prevent their exploitation (virtual patching). Therefore, even the best bounty will never replace your WAF.

2. Bug Bounty is not suitable to test private and sensitive applications

Many talented researchers, listed in numerous Halls of Fame, live in developing countries. So, how comfortable are you in sending a 2FA token for your production e-baking web portal to Siberia or Bangalore? Even acting via a managed Bug Bounty programme you will never achieve the same level of financial insurance, liability and personnel clearance as you may expect from a professional pen-testing company.

3. You cannot compensate quality by quantity

While some people in the industry have argued that open-source software is more secure than proprietary code because anyone can check the code for security flaws and weaknesses. Some of those have changed their opinion after vulnerabilities like Heartbleed appeared on the horizon, putting into question the entire security concept of open-source. In information security, you cannot compensate quality of testing by quantity.

Bug Bounty management companies can restrict your bounty project to researchers with recordable reputation or specific skills. However, they just shift their business models towards classical penetration testing with unclear profitability, as the more limits you introduce, the more difficult it is to find qualified researchers to work remotely and get paid by results.

Vulnerability testing methodologies should be proportional to your web infrastructure size, scope and, most important, its expected usage in production. If you are a large company with numerous publicly facing web applications available and designed for everyone – a bounty programme can definitely be a great added-value for your security arsenal. Otherwise, you'd be better spending your budget on traditional web security solutions, proper integration of S-SDLC, and mitigation of the most popular web application security risks.

Contributed by Ilia Kolochenko, CEO of High-Tech Bridge