Passive DNS Daily Files Endpoint
The Passive DNS Daily Files Endpoint allows a user to download dumps of passive DNS daily files using HTTPS. These files are in a CSV format and are compressed using gzip. The previous days files are available by 00:30 UTC.
Supported Files
Address IPv4
The address IPv4 files contain data of hostname and IPv4 pairs with the epoch timestamp of the query.The IP is in the form of a long integer. The filename is in the format of ‘address_ipv4_YYYYMMDD.csv.gz’ where ‘YYYYMMDD’ is the format of the date of the file. The contents of the file are in the format below:
timestamp | ipv4 | hostname
Address IPv6
The address IPv6 files contain data of hostname and IPv6 pairs with the epoch timestamp of the query.The IP is in a hex format. The filename is in the format of ‘address_ipv6_YYYYMMDD.csv.gz’ where ‘YYYYMMDD’ is the format of the date of the file. The contents of the file are in the format below:
timestamp | ipv6 | hostname
CNAME
The cname files contain data of canonical hostname and hostname pairs with the epoch timestamp of the query. The filename is in the format of ‘cname_YYYYMMDD.csv.gz’ where ‘YYYYMMDD’ is the format of the date of the file. The contents of the file are in the format below:
timestamp | canonical hostname | hostname
MX
The mx files contain data of domain and mx pairs (priority included) with the epoch timestamp of the query. The filename is in the format of ‘mx_YYYYMMDD.csv.gz’ where ‘YYYYMMDD’ is the format of the date of the file. The contents of the file are in the format below:
timestamp | domain | mx
Nameserver
The nameserver files contain data of domain and nameserver pairs with the epoch timestamp of the query. The filename is in the format of ‘mx_YYYYMMDD.csv.gz’ where ‘YYYYMMDD’ is the format of the date of the file. The contents of the file are in the format below:
timestamp | domain | nameserver
New Domains
The new domains files contain data of newly seen domains with the first seen epoch timestamp of the query. The filename is in the format of ‘new_domains_YYYYMMDD.csv.gz’ where ‘YYYYMMDD’ is the format of the date of the file. The contents of the file are in the format below:
timestamp | domain
NBD
The NBD files contain data of newly seen domains with only domains that match the criteria outlined here https://docs.spamhaus.com/sia/docs/source/02-data-explained/data-anatomy.html#domain-address-records, with the first seen epoch timestamp of the query. The filename is in the format of ‘nbd_YYYYMMDD.csv.gz’ where ‘YYYYMMDD’ is the format of the date of the file. The contents of the file are in the format below:
timestamp | domain
Example Script
Below is an example script for downloading the files:
#!/bin/sh
USER=user
DATADIR=/tmp
PASSWORD=password
BASE_URL=https://daily-01.deteque.com/pdns/
DATE=`date --date="yesterday" "+%Y%m%d"`
/usr/bin/curl --silent --user ${USER}:${PASSWORD} ${BASE_URL}/address_ipv4_${DATE}.csv.gz -o ${DATADIR}/address_ipv4_${DATE}.csv.gz
/usr/bin/curl --silent --user ${USER}:${PASSWORD} ${BASE_URL}/address_ipv6_${DATE}.csv.gz -o ${DATADIR}/address_ipv6_${DATE}.csv.gz
/usr/bin/curl --silent --user ${USER}:${PASSWORD} ${BASE_URL}/cname_${DATE}.csv.gz -o ${DATADIR}/cname_${DATE}.csv.gz
/usr/bin/curl --silent --user ${USER}:${PASSWORD} ${BASE_URL}/mx_${DATE}.csv.gz -o ${DATADIR}/mx_${DATE}.csv.gz
/usr/bin/curl --silent --user ${USER}:${PASSWORD} ${BASE_URL}/nameserver_${DATE}.csv.gz -o ${DATADIR}/nameserver_${DATE}.csv.gz
/usr/bin/curl --silent --user ${USER}:${PASSWORD} ${BASE_URL}/nbd_${DATE}.csv.gz -o ${DATADIR}/nbd_${DATE}.csv.gz
/usr/bin/curl --silent --user ${USER}:${PASSWORD} ${BASE_URL}/new_domains_${DATE}.csv.gz -o ${DATADIR}/new_domains_${DATE}.csv.gz