A URL scanning service included in products from major security vendors has been leaking personal and sensitive information from links to corporate web pages that could be gold for threat actors.
This revelation comes from a report by researchers at Positive Security.
Part of the problem, which was quietly fixed in July, was in urlscan.io, used by a number of firewall and gateway products that link to the service for scanning and analyzing URLs in emails and websites. Part of the problem is also poorly-created URLs that include sensitive information.
But misconfigured security orchestration, automation and response (SOAR) tools in applications that create databases of URLs also play a role. If left open to the internet and found by the wrong person, these databases can be mined for their contents.
These include links to password reset pages, account creation pages, API keys, DocuSign signing request pages, Sharepoint invites, WebEx meeting recording pages, PayPal invoices, and team meeting invites.
“Overall, the urlscan.io service contains a trove of sensitive information of various kinds, that can be used by hackers, spammers, or cyber criminals, for example to take over accounts, perform identity theft, or run believable phishing campaigns,” says the report.
According to a report, urlscan.io lists 26 commercial security solutions by vendors such as Palo Alto, Splunk, Rapid7, FireEye and ArcSight that have integrated the service via its API. Others, like GitHub, use the urlscan.io API internally.
“If any of those tools/API users are accidentally performing public URL scans, this could lead to systematic data leakage,” says the report. “As those advanced security tools are mostly installed in large corporations and government organizations, leaked information could be particularly sensitive.”
Besides commercial products, the report says the integration page also lists 22 open source projects, some of which are information-gathering tools, and others are simple library implementations for easier querying of the API.
Scans done by urlscan.io can include a lot of information, says the report, including:
- the submitted URL (with all GET parameters);
- the effective URL in case of a redirect;
- any HTTP requests that were performed while scraping/scanning the URL;
- information about the IPs and domains communicated with;
- a screenshot of the page taken at the time of the scan;
- the full HTML response of the site.
Organizations inadvertently create vulnerable databases of scanned URLS through the Security Orchestration, Automation and Response (SOAR) capabilities of the security platforms they use, the report says. SOAR allows organizations to write their own playbooks to connect different data sources with security tools and services. To ease development, the platforms offer integrations with several third-party services such as urlscan.io. With urlscan.io’s pack installed, a playbook could extract URLs from incoming emails and submit them to urlscan.io with an automated command.
However, under certain conditions, such as a mistake in a playbook, a misconfiguration of the urlscan.io integration or account visibility settings, or if the integration itself has a bug that does not respect the user-chosen visibility, a scan can be wrongly submitted as public.
After being tipped off, urlscan.io released a new version in July, with added deletion rules to periodically delete scan results matching certain search patterns. It highlights the default visibility setting in its user interface, and adds an option to set a team-wide maximum visibility.
It also published a blog post titled “Scan Visibility Best Practices” that explains scan visibility settings, encourages users to frequently review their submissions, and details urlscan’s efforts to prevent such leaks.
Web services should make sure that they are expiring password reset and similar links quickly and are not leaking unnecessary information to unauthenticated users via links that could become public, the report says. On an unsubscribe page, redact the user’s email address, and ask for additional authentication/information before showing personal information. For example, many package tracking websites now ask for a ZIP code before showing the full address. When implementing API authentication, the report adds, don’t accept API keys via GET parameters; instead, require the use of a separate HTTP header.
The report says IT administrators can also search urlscan.io as well as other services for any data leaks regarding their own web service or organization, request deletions/exclusions, and, if necessary, disable and rotate any leaked API keys of users.
urlscan.io users/security teams that integrate the service should review their command, integration and account visibility settings, adds the report, keep their integrations up to date, regularly review their submitted scans, and check the urlscan.io blog post for more information.
