Webv2 - HTTP(s) scanning revamped
For the longest time there has been one common request that we've gotten from the users of our platform.
"Can we have hashes of favicons to search for common websites?"
And for the longest time we have been working on it, however we ended up deciding that it needed to be part of a larger revamp that we wanted to do with our HTTP(s) modules.
As of this month we are gonna be deprecating the HTTP and HTTPS modules. Both of these allowed customers to create fully customized HTTP requests that they could use on ondemand scans against IPs or domains at large scale.
New modules on the block - Webv2 and Web Enrich
As of today, two new modules have been added.
Webv2 is now the module used for our internet-wide scans, you will immediately be able to look at favicon data, and click on it when doing a search to see similar websites.
Two hashes exist, md5 and mmh3 so it makes life easier to interplay with other tooling you might use today.
If you are an enterprise customer you now also have the ability to get the response content and screenshots of pre-javascript render and post-javascript render by using this module for on-demand scans.
If you're one of our customers using BinaryEdge to do large scale domain analysis it should be a feature extremely useful for you.
You also get ssdeep hashes automatically generated for you, when looking at phishing and trying to compare similarity at large scale this should be something super useful.
This brings us to Web Enrich - this module won't be run at the moment at internet wide scale but rather be a tool for enterprise customers that want to extract extra-metadata from web frameworks of the targets (IP addresses or domains) they are analysing. Web enrich is constantly being upgraded with new ways of extracting metadata without any type of exploitation or authentication on different web frame works.
Here are some examples of its findings:
Wordpress
{
...
"result": {
"data":{
"url":"https://URL/",
"name":"WordPress",
"category":[
"CMS",
"Blogs"
],
"wordpress":{
"version":[
"4.9.13"
],
"cpe":"cpe:/a:wordpress:wordpress:4.9.13",
"confidence":100,
"plugins":[
{
"name":"WP Social Icons",
"slug":"wp-social-icons",
"version":"1.1",
"site_url":"https://URL/wp-content/plugins/wp-social-icons/readme.txt",
"plugin_url":"https://plugins.svn.wordpress.org/wp-social-icons/"
},
{
"name":"Contact Form 7",
"slug":"contact-form-7",
"version":"4.4.1",
"site_url":"https://URL/wp-content/plugins/contact-form-7/readme.txt",
"plugin_url":"https://plugins.svn.wordpress.org/contact-form-7/"
}
],
"themes":[
{
"name": "RT-Theme 17",
"slug": "rttheme17",
"version": "2.7.1",
"site_url": "http://URL/wp-content/themes/rttheme17/style.css",
"theme_url": "http://themeforest.net/user/stmcan"
}
],
"users":[
{
"name":"axxxxn",
"username":"axxxxn",
"email_md5":"43xxxxxxxxxxxxxxxxxxxxxxxxxxxx11"
},
{
"name":"Axxxxxxxxxxxxk",
"username":"axxxxxa",
"email_md5":"59xxxxxxxxxxxxxxxxxxxxxxxxxxxxbf"
},
{
"name":"Lxxxxxxxxxxo",
"username":"lxxxxxe",
"email_md5":"13xxxxxxxxxxxxxxxxxxxxxxxxxxxx30"
}
],
"directory_listing":[
"/wp-includes/",
"/wp-content/uploads/"
],
"internal_information": {
"internal_path":"/home/naxxxxxor/public_html/"
}
},
"http_version":"HTTP/1.1",
"ssl":true,
"fqdn":"URL",
"headers":{
"date":"Wed, 12 Feb 2020 01:15:43 GMT",
"content-type":"text/html; charset=UTF-8",
"transfer-encoding":"chunked",
"connection":"keep-alive",
"set-cookie":"__cfduid=d3f69fd24fe38afa5984d479c144382501581470142; expires=Fri, 13-Mar-20 01:15:42 GMT; path=/; domain=.xxxx.com; HttpOnly; SameSite=Lax",
"x-powered-by":"PHP/7.2.27",
"x-pingback":"https://URL/xmlrpc.php",
"link":"<https://URL/wp-json/>; rel=\"https://api.w.org/\", <https://URL/>; rel=shortlink",
"cache-control":"max-age=600",
"expires":"Wed, 12 Feb 2020 01:25:43 GMT",
"vary":"Accept-Encoding,User-Agent",
"strict-transport-security":"max-age=31536000",
"cf-cache-status":"DYNAMIC",
"expect-ct":"max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
"server":"cloudflare",
"cf-ray":"563ab6897a2c546c-MAD",
"content-encoding":"gzip"
},
"redirects":[
{
"status_code":301,
"redirect_uri":"http://IP:80/",
"headers":{
"date":"Wed, 12 Feb 2020 01:15:41 GMT",
"server":"Apache",
"x-powered-by":"PHP/7.2.27",
"x-pingback":"http://URL/xmlrpc.php",
"location":"https://IP/",
"cache-control":"max-age=600",
"expires":"Wed, 12 Feb 2020 01:25:41 GMT",
"strict-transport-security":"max-age=31536000",
"vary":"User-Agent",
"content-length":"0",
"keep-alive":"timeout=5, max=100",
"connection":"Keep-Alive",
"content-type":"text/html; charset=UTF-8"
}
},
{
"status_code":301,
"redirect_uri":"https://IP/",
"headers":{
"date":"Wed, 12 Feb 2020 01:15:42 GMT",
"server":"Apache",
"x-powered-by":"PHP/7.2.27",
"x-pingback":"https://URL/xmlrpc.php",
"location":"https://URL/",
"cache-control":"max-age=600",
"expires":"Wed, 12 Feb 2020 01:25:42 GMT",
"strict-transport-security":"max-age=31536000",
"vary":"User-Agent",
"content-length":"0",
"keep-alive":"timeout=5, max=100",
"connection":"Keep-Alive",
"content-type":"text/html; charset=UTF-8"
}
}
]
}
}
}We currently extract metadata from Wordpress, Umbraco, Joomla,Magento,Citrix, F5 and detect secrets such as amazon S3 URLs, and Square oauth tokens.
To learn more about our Enterprise features and request a trial, please contact [email protected]
As with any other modules, if you have suggestions of other web frameworks for us to add or metadata we can extract please feel free to ping us on slack or twitter!
Also, WE ARE HIRING!