Yahoo launches a new open-source project that will allow system administrators to perform URL scanning in order to find the presence of common security vulnerabilities and malicious web content.
Dubbed "Project Gryffin," it will initially launch in beta and will be under the BSD-style license, the type of license that Yahoo likes to use whenever it is launching open source projects.
According to Yahoo, Gryffin is a large-scale Web security scanning platform designed to address two specific issues namely Coverage and Scale.
Coverage is said to be made up of two dimensions. The first one, crawl, refers to having the ability to find as much of the application's footprint as possible. The second dimension, scan, refers to having the ability to test each application part based on applied set of vulnerabilities.
Yahoo said that while Gryffin is released as a standalone package, in reality, it is built for scale under the publisher-subscriber model. Since the platform has components of either publishers or subscribers or a combination of both, it is therefore allowed to scale horizontally simply by adding new nodes to it.
Other features of crawl include a deduplication engine meant to avoid crawling the same page two times by comparing new and existing pages and PhantomJS which is designed to handle DOM rendering in JavaScript based applications.
Gryffin's requirements include: Go, NSQ distributed messaging system, PhantomJS v2, Arachni for scanning Web vulnerabilities and XSS, Sqlmap for scanning SQL injection, and Kibana and Elastic Search for dashboarding.
"It's not wise to reinvent the wheel where you do not have to," explains Yahoo. "Gryffin at production scale at Yahoo uses open source and custom fuzzers. Some of these custom fuzzers might be open sourced in future and might or might not be part of Gryffin repository."
Apart from Gryffin, Yahoo had already launched a number of open-source projects in the past which were at first seen as strange but were eventually adopted by a number of industry players. Some of these projects are identified as Hadoop, Pure and YUI!
Yahoo also wanted to use the platform in order to achieve an elastic infrastructure and wider coverage.
"Better coverage translates to fewer false negatives," said the team from Yahoo. "Inherent scalability translates to capability of scanning, and supporting a large elastic application infrastructure. Simply put, the ability to scan 1000 applications today to 100,000 applications tomorrow by straightforward horizontal scaling."
