This week Verizon Communications announced its Oath division — which owns Yahoo, AOL and other media services — had paid out US$5 million in bug bounties this year.
That’s five times more than it paid out in 2017.
Clearly there’s money in vulnerabilities, and not just for criminals.
With no developer or developer team able to assure an organization that it can churn out perfectly secure code, bug bounty programs and the money paid for those who find bugs will only go up in the foreseeable future.
This year HP Inc. added a private, invitation-only bug bounty program for its printer division with up to US$10,000 available for every serious vulnerability found. GitLab, an open source DevOps platform, has made its private bounty program open to any ethical hacker, with prizes of up to US$12,000 for critical vulnerabilities. Facebook expanded its bug bounty program to add rewards for finding vulnerabilities that involve the exposure of user access tokens.
In May, Google paid a teenager from Uruguay more than US$36,000 for a bug.
UPDATE: On Wednesday the U.S. Congress passed a bill tp establish bug bounty and vulnerability disclosure programs at the Department of Homeland Security.
Bounty programs are so useful that some organizations hold hackathons. In October the U.S. Defense Department backed a Hack the Marines Corps contest with 100 invited hackers to test the Leathernecks’ public-facing websites and services. A total of US$150,00 in prizes was available.
In November several hackers walked off with a total of US$225,000 in prizes on the first day of the Tokyo Pwn2Own conference, one of several Pwn2Own conferences held around the world.
Shopify, an e-commerce platform based in Ottawa, has held two live hacking events. The first in February 2017 paid a total $42,000 to hackers/researchers, while the second, this past October paid out $116,000.
Among the advantages of a short-term hackathon is a bunch of vulnerabilities can be found at once.
For white hat hackers and security researchers, money and reputation are only part of the reasons to participate. “I love it … it’s an analytical challenge,” says Peter Yaworski, a Toronto-based member of Shopify’s application security team who goes vulnerability hunting in his spare time — that is, five nights a week after his kids have gone to bed.
“There’s always going to be bugs in software,” he explained in an interview, “so you know something’s out there. It’s just a question of can you find it.”
“You kind of want to assume you’re smarter than the developer,” he added.
Before we go further, a definition: A bounty program pays money for vulnerabilities. A vulnerability disclosure program is merely a mechanism an organization has for reporting bugs to its security team.
Some firms, particularly technology companies, run their own bug bounty programs. Most use services set up by companies like Bugcrowd (customers include HP and Mastercard), HackerOne (customers include Shopify, General Motors) or Synack (customers include the U.S. Defense Department).
‘Crowdsourced security’
Sometimes calling themselves crowdsourced security, they recruit ethical hacker/researchers, offer platforms for organizing the reporting and rating and screening the ethical hackers. HackerOne says it has 200,000 hackers registered and rated on the number and quality of the bugs they find. For customers who have high security needs, HackerOne will run security checks on hackers who are willing.
Often an organization will start with a private program, with a limited number of hackers invited to hunt for bugs. The organization sets the boundaries (ie. we only want you to look for bugs on our web sites) and the price it will pay for a vulnerability.
HP Inc., which formally announced its private printer bug bounty program on July 31, went with Bugcrowd. The idea, explained Shivaun Albright, HP’s chief technologist for print security, is to add to the company’s existing software development security processes for printers.
“What we want to do is have it [the program] look for some of these obscure defects that could be exploited on our devices,” she said. The program was officially announced after a three-month pilot. “Thus far we find the program has been very successful.”
HP has invited about 35 of the roughly 50,000 Bugcrowd hackers into its program, she said. At any time there are about 15 printers available for them to try to crack.
“The huge benefit we see is it gives us access to hacking, reverse-engineering, penetration skills that are difficult to find in the industry” for finding vulnerabilities full-time developers have missed, Albright said. Lessons learned can be incorporated into HP’s development processes.
Shopify, which is headquartered in Ottawa, is a web-based e-commerce platform for small and medium businesses. Coincidentally, its choice of HackerOne for its bug bounty program led to Yaworski’s hiring.
In 2015 he was earning enough as a hacker for the platform that he was among a number of people invited to a contest HackerOne was running in San Francisco. That led to a job offer from Shopify.
Shopify has two programs on HackerOne; a Core program, which pays for bugs found on the main platform and apps and has so far paid out $369,700; and Scripts, which pays for vulnerabilities found in Shopify’s implementation of mruby on its platform. Mruby is a lightweight interpreter for the Ruby programming language. The Scripts program has so fair paid out about $594,900
One tech company that runs its own bug bounty program is Japan’s Trend Micro. Called the Zero Day Initiative (ZDI), the company believes it’s the world’s largest vendor-agnostic program which pays out for bugs found a number of systems, in addition to its own. These include products from Apple, Adobe, Microsoft and Google. It recently added open source products like Drupal, Jumla and WordPress.
According to a Frost and Sullivan report, over 66 per cent of major vulnerabilities found in 2017 came from the ZDI program. Google’s Project Zero was second.
Dustin Childs, the ZDI communications manager, said in an interview the program paid out over US$2 million in 2017. In the first 6 months of this year it paid out over US$1 million for about 470 vulnerabilities, he expects this year it will pay out more than it did last year.
Practically anyone can register with the program to be a hacker, except those from certain countries. Those accepted get an encrypted key through which they can send a bug report. If ZDI researchers confirm it’s legit, a payment is offered. The hacker can negotiate the value. If the two sides can’t come to an agreement, the researcher can try elsewhere. If ZDI accepts the bug is reported to the vendor, with Trend Micro helping to create a patch. Vendors have 120 days to release a patch, after which ZDI releases information about the bug. That, Childs explained, prevents vendors from “sweeping it under the rug.” Hacker/researchers gets the credit and can talk publicly about their work if they want.
What Trend Micro gets out of the program is threat intelligence on bugs that it can build detection filters into the endpoint and network security products it sells.
Targeted applications
ZDI was updated this year with a targeted incentive program to get researchers to look closer at what it calls “high profile targets,” such as Jumla, Drupal, WordPress, Apache HTPPS server and Microsoft’s IIS server. This program runs like a contest, with new targets announced periodically. Prizes are higher than average, with the first to submit a verified exploit winning big bucks — currently, US$200,000 is being offered for an exploit for the latest versions of Ubuntu or Windows server.
Backers of bug bounty programs maintain they are excellent ways to use outside skilled talent find holes in products. However, some say they endanger end users when a white hat hacker publicly discloses a vulnerability because they aren’t satisfied a security update will be released, or if a vendor decides not to release a fix. Even if a supposed bug is small, a vendor’s reputation could be damaged by a news story alleging a product has a vulnerability. That in part is why politicians in some jurisdictions are talking about making it a crime for researchers to divulge a bug.
“That’s a little silly,” said Adam Baccus, director of program operations at HackerOne. It would be like trying to ban locksmiths as a profession, he said. Bug bounty programs are about understanding the safe rules for vulnerability searching.
With over 200,000 registered hackers, HackerOne helps customers with setting up their programs, such as setting rules of engagement (what researchers are allowed to do, what they can/cannot hack, payment scale) and ways to evaulate submitted bugs.
Over the years it has paid out more than US$32 million for finding and fixing more than 76 million vulnerabilities.
Got to act
There are also ethical issues with having a bounty or vulnerability reporting program. “When someone comes to [a company] with a bug and says they’ve found something, in not in all instances have we seen software companies sit up and take note,” says Tony Anscombe, global security evangelist at security vendor ESET. “It’s important you take t seriously and you get it fixed.”
ZDI’s Childs feels too many companies think if they offer a reward for vulnerabilities they will actually be told of bugs and then have to do something. “But I caution companies to make sure their response programs are mature enough to act on submissions they receive before starting. It’s certainly not a panacea and not for everyone. What I tell manufactures is its always cheaper to find bugs before releasing products than pay for them afterwards.”
Running a program yourself versus paying a platform are matters of affordability and convenience. But, adds Childs, at least you should have a response process to fix bugs.
Bug bounty programs also raise the interesting question of why they are needed. When will developers smarten up and reduce the number of vulnerabilities in their code?
Not for a while, suggests Baccus. While developers at some companies are getting better — and in some cases that has meant they have raised the price of their bounties — cross-site scripting vulnerabilities on websites have been a problem for years, he noted, and still hasn’t been eliminated. “I don’t think HackerOne will ever go out of business. More and more organizations are spinning up, more startups are coming out of nowhere and more people are developing software and code.”
Dual value
Bug bounty programs have a dual value, he added: Not only do bugs get fixed, they also are a source of information to developers on why they happened in the first place. Many IT teams perform a root cause analysis on being told of a vulnerability to discover why it fell through the cracks, Baccus noted. In some cases they can look at years of data to figure out broader trends (50 per cent of our bugs come from this department. Maybe it’s a security training problem.)
While a rewards program provides great value, Yaworski also cautions it isn’t a silver bullet. “Having a bounty program isn’t something that’s going to make your app secure. You need to have buy-in from the entire company, including the security team” to ensure bugs are fixed. He also warned the program won’t work unless there is consistency in payouts and with support staff willing to deal with ethical hackers/researchers. Lacking that, an organization will alienate the hackers and in-house developers.
Management shouldn’t blame developers when bugs are reported, he adds — there will always be bugs. The main issue is how the internal development process failed. “If you take that position, the goal should be to eliminate as many [vulnerabilities] as you can, while balancing continuing need to ship code and develop your application. But at the same time when a bug comes in, the goal should be you don’t introduce the same bug twice” when it’s fixed.
“Don’t make app security an afterthought,” Yaworski urges organizations. “It has to be baked in at the beginning. That means hiring security-conscious developers, and if you can’t find them pay to train the ones you have.”
