Deteque Secure Feeds

Deteque offers Passive DNS feeds based on a Kafka distribution model. The feeds consist of multiple “topics”, each representing a single type of passive data set. Each data pair includes an epoch timestamp. The current set of topics available via this delivery mechanism are:

Topic
• spamhaus_new_domains
• spamhaus_pdns_cname
• spamhaus_pdns_ipv4
• spamhaus_pdns_ipv6
• spamhaus_pdns_mx
• spamhaus_pdns_ns
• spamhaus_pdns_txt
• spamhaus_pdns_soa

spamhaus_new_domains are domains that have been seen for the first time in passive dns feeds. The data format for this topic is a pipe separated string consisting of:

NS|timestamp|domain|nameserver

spamhaus_pdns_cname consists of CNAME records. The data format for this topic is a pipe separated string consisting of:

timestamp|host|cname

spamhaus_pdns_ipv4 consists of host/ipv4 data pairs. The data format for this topic is a pipe separate string consisting of:

timestamp|hostname|ipv4 address

spamhaus_pdns_ipv6 consists of host/ipv6 data pairs. The data format for this topic is a pipe separate string consisting of:

timestamp|hostname|ipv6 address

spamhaus_pdns-mx consistsof MX records. The data format for this topic is a pipe separated string consisting of:

timestamp|host/domain|mx record

spamhaus_pdns_ns consist of nameserver/domain data pairs. The data format for this topic is a pipe separated string consisting of:

timestamp|domain|nameserver

spamhaus_pdns_txt is in a json format with the looked up host under “Hdr” as “Name” and the returned TXT record in an array:

{ 
 "Hdr": { 
   "Epoch": 1641397070, 
   "Name": "example.com.", 
   "Rrtype": 16, 
   "Class": 1, 
   "Ttl": 60, 
   "Rdlength": 35 
 }, 
 "Txt": ["v=spf1 -all"] 
}

spamhaus_pdns_soa is in a json format with the looked up host under “Hdr” as “Name” and the returned SOA information in named fields:

{ 
 "Hdr": { 
   "Epoch": 1641397849, 
   "Name": "example.com.", 
   "Rrtype": 6, 
   "Class": 1, 
   "Ttl": 3601, 
   "Rdlength": 58 
 }, 
 "Ns": "ns.icann.org.", 
 "Mbox": "noc.dns.icann.org.", 
 "Serial": 2021120718, 
 "Refresh": 7200, 
 "Retry": 3600, 
 "Expire": 1209600, 
 "Minttl": 3600 
}

VPN Access


A VPN is required to access the Kafka distribution servers. Our VPN is based on Wireguard (https://www.wireguard.com/), a free and open source product that provides extremely efficient encryption with excellent throughput.
Wireguard is actually built into Linux kernels 5.4 and later. If you’re using an older kernel, a Golang based program will be required. A minimalist Wireguard installation only requires a single config file that’s consists of less than 10 lines. On Linux, the configuration file is located at /etc/wireguard/wg0.conf
Deteque will provide each customer with the wg0.conf file that they’ll need to securely access the Kafka back end. The Wireguard project provides a link that explains how to install Wireguard under a variety of different platforms. That link is located at: https://www.wireguard.com/install/
Using Debian Bullseye or Ubuntu 18+ all that’s needed to install Wireguard is “apt install wireguard wireguard-tools”.
Each set of Wireguard credentials can only be used on one server. Any customer requiring access to the Kafka feeds from more than one server will require additional Wireguard credentials. Due to the volume of data customers are not permitted to download multiple copies of feeds.

Client Access To Passive DNS Feeds


Once the Wireguard VPN is installed and configured you’ll need a Kafka client installed and configured to connect the the Kafka servers. There are a number of Kafka clients available for a myriad of programming languages so you should choose what works best for you.
Deteque will provide each customer a unique “consumer group” for your use which will track the last message read.
For examples of how to program a kafka client, we’ll provide three different clients - one running php, one running python and one running Golang. All clients basically need to successfully connect are:
- Customer User name (assigned by Deteque)
- Customer Password (assigned by Deteque)
- List of “brokers” (server IPs to connect to)
- Consumer group (topic)