Deteque Secure Feeds

Deteque offers Passive DNS feeds based on a Kafka distribution model. The feeds consist of multiple “topics”, each representing a single type of passive data set. Each data pair includes an epoch timestamp. The current set of topics available via this delivery mechanism are:

Topic
spamhaus_new_domains
spamhaus_nbd
spamhaus_pdns_cname
spamhaus_pdns_ipv4
spamhaus_pdns_ipv6
spamhaus_pdns_mx
spamhaus_pdns_ns
spamhaus_pdns_txt
spamhaus_pdns_soa
spamhaus_chaff_cname
spamhaus_chaff_ipv4
spamhaus_chaff_ipv6
spamhaus_chaff_ns

spamhaus_new_domains contains data of newly seen domains. The data format for this topic is a pipe separated string consisting of:

NS|timestamp|domain|nameserver

spamhaus_nbd contains data of newly seen domains with only domains that match the criteria outlined here https://docs.spamhaus.com/sia/docs/source/02-data-explained/data-anatomy.html#domain-address-records.The data format for this topic is a pipe separated string consisting of:

timestamp|domain

spamhaus_pdns_cname consists of CNAME records. The data format for this topic is a pipe separated string consisting of:

timestamp|host|cname

spamhaus_pdns_ipv4 consists of host/ipv4 data pairs. The data format for this topic is a pipe separate string consisting of:

timestamp|hostname|ipv4 address

spamhaus_pdns_ipv6 consists of host/ipv6 data pairs. The data format for this topic is a pipe separate string consisting of:

timestamp|hostname|ipv6 address

spamhaus_pdns_mx consistsof MX records. The data format for this topic is a pipe separated string consisting of:

timestamp|host/domain|mx record

spamhaus_pdns_ns consist of nameserver/domain data pairs. The data format for this topic is a pipe separated string consisting of:

timestamp|domain|nameserver

spamhaus_pdns_txt is in a json format with the looked up host under “Hdr” as “Name” and the returned TXT record in an array:

{ 
 "Hdr": { 
   "Epoch": 1641397070, 
   "Name": "example.com.", 
   "Rrtype": 16, 
   "Class": 1, 
   "Ttl": 60, 
   "Rdlength": 35 
 }, 
 "Txt": ["v=spf1 -all"] 
}

spamhaus_pdns_soa is in a json format with the looked up host under “Hdr” as “Name” and the returned SOA information in named fields:

{ 
 "Hdr": { 
   "Epoch": 1641397849, 
   "Name": "example.com.", 
   "Rrtype": 6, 
   "Class": 1, 
   "Ttl": 3601, 
   "Rdlength": 58 
 }, 
 "Ns": "ns.icann.org.", 
 "Mbox": "noc.dns.icann.org.", 
 "Serial": 2021120718, 
 "Refresh": 7200, 
 "Retry": 3600, 
 "Expire": 1209600, 
 "Minttl": 3600 
}

spamhaus_chaff_cname consists of CNAME records that have been determined to be chaff data. Chaff is data that does not pertain to DNS, with attributes such as session data, CDN mappings, and other abusive qualities in the hostname. The data format for this topic is a pipe separated string consisting of:

timestamp|host|cname

spamhaus_chaff_ipv4 consists of host/ipv4 data pairs that have been determined to be chaff data. Chaff is data that does not pertain to DNS, with attributes such as session data, CDN mappings, and other abusive qualities in the hostname. The data format for this topic is a pipe separate string consisting of: timestamp|hostname|ipv4 address

spamhaus_chaff_ipv6 consists of host/ipv6 data pairs that have been determined to be chaff data. Chaff is data that does not pertain to DNS, with attributes such as session data, CDN mappings, and other abusive qualities in the hostname. The data format for this topic is a pipe separate string consisting of:

timestamp|hostname|ipv6 address

spamhaus_chaff_ns consist of nameserver/domain data pairs that have been determined to be chaff data. Chaff is data that does not pertain to DNS, with attributes such as session data, CDN mappings, and other abusive qualities in the hostname. The data format for this topic is a pipe separated string consisting of:

timestamp|domain|nameserver

VPN Access

A VPN is required to access the Kafka distribution servers. Our VPN is based on Wireguard (https://www.wireguard.com/), a free and open source product that provides extremely efficient encryption with excellent throughput.
Wireguard is actually built into Linux kernels 5.4 and later. If you’re using an older kernel, a Golang based program will be required. A minimalist Wireguard installation only requires a single config file that’s consists of less than 10 lines. On Linux, the configuration file is located at /etc/wireguard/wg0.conf
Deteque will provide each customer with the wg0.conf file that they’ll need to securely access the Kafka back end. The Wireguard project provides a link that explains how to install Wireguard under a variety of different platforms. That link is located at: https://www.wireguard.com/install/
Using Debian Bullseye or Ubuntu 18+ all that’s needed to install Wireguard is “apt install wireguard wireguard-tools”.
Each set of Wireguard credentials can only be used on one server. Any customer requiring access to the Kafka feeds from more than one server will require additional Wireguard credentials. Due to the volume of data customers are not permitted to download multiple copies of feeds.
Once Wireguard is installed run this command to set it up as a service so it will start automatically at boot time:

systemctl enable wg-quick@wg0.service
systemctl daemon-reload

To stop or start the Wireguard tunnel either use systemd or run the commands below

Start:
/usr/bin/wg-quick up wg0

Stop:
/usr/bin/wg-quick down wg0

We recomend you put these commands into scripts for ease of use.

Once the Wireguard tunnel is activate you should be able to ping the Kafka/Redpanda servers

ping 10.131.0.11
ping 10.131.0.12
ping 10.131.0.13
ping 10.131.0.14

Client Access To Passive DNS Feeds

Once the Wireguard VPN is installed and configured you’ll need a Kafka client installed and configured to connect the the Kafka servers. There are a number of Kafka clients available for a myriad of programming languages so you should choose what works best for you.
Deteque will provide each customer a unique “consumer group” for your use which will track the last message read.
For examples of how to program a kafka client, we’ll provide three different clients - one running php, one running python and one running Golang. All clients basically need to successfully connect are:
- Customer User name (assigned by Deteque)
- Customer Password (assigned by Deteque)
- List of “brokers” (server IPs to connect to)
- Consumer group (topic)