Passive recon using public datasets

There are projects that gather Internet wide scan data and make it available to researchers and the security community. The datasets published by this projects are a treasure trove of sub-domain information. Although finding sub-domains in this massive datasets is like finding a needle in the haystack, it is worth the effort.

Following are few public datasets that aggregate information that could be of interest during sub-domain enumeration:

Name Description Price
Sonar FDNS, RDNS, UDP, TCP, TLS, HTTP, HTTPS scan data FREE
Censys.io TCP, TLS, HTTP, HTTPS scan data FREE
CT TLS FREE
CZDS DNS zone files for "new" global TLDs FREE
ARIN American IP registry information (ASN, Org, Net, Poc) FREE
CAIDA PFX2AS IPv4 Daily snapshots of ASN to IPv4 mappings FREE
CAIDA PFX2AS IPv6 Daily snapshots of ASN to IPv6 mappings FREE
US Gov US government domain names FREE
UK Gov UK government domain names FREE
RIR Delegations Regional IP allocations FREE
PremiumDrops DNS zone files for com/net/info/org/biz/xxx/sk/us TLDs $24.95/mo
WWWS.io Domains across many TLDs (~198m) $9/mo
WhoisXMLAPI.com New domain whois data $109/mo

Source - https://github.com/fathom6/inetdata

Discovering sub-domains using Rapid7 Forward DNS dataset

Forward DNS dataset is published as part of Project Sonar. This data is created by extracting domain names from a number of sources and then sending an ANY query for each domain. The data format is a gzip-compressed JSON file. We can parse the dataset to find sub-domains for a given domain. The dataset is massive though(20+GB compressed, 300+GB uncompressed)

# Command to parse & extract sub-domains for a given domain
$ curl -silent https://scans.io/data/rapid7/sonar.fdns_v2/20170417-fdns.json.gz | pigz -dc | grep “.icann.org” | jq

fdns-search

results matching ""

    No results matching ""