Open source IP blacklist checker tool

By on 17 Apr 2024

Category: Tech matters

Tags: , , , , ,

Blog home

The AbuseBox landing page.

At Unitel LLC, we have developed an IP blacklist-checking tool, called AbuseBox, designed to monitor and verify IP addresses against blacklists, with the capability to submit delisting requests. Our research indicates that approximately 40% of all addresses find themselves on such blacklists, underscoring the importance of addressing this issue within our economy.

This post describes our basic goals for AbuseBox and how to use it.

Why are IP addresses blacklisted?

A blacklist is a method of creating a registry by detecting abnormal behaviour of users connected to the Internet on many sensors located around the world. Suppose suspicious behaviour is detected on the Internet. In that case, the sensors send a warning to the ISP or the address owner, and if the suspicious behaviour is repeated several times, the address is blacklisted.

Abnormal behaviours can include downloading or distributing copyright-violating or illegal content, intrusion attempts — sending spam mail or malicious traffic from a user’s device infected with a virus, attempting to attack other networks, and so on.

Risks of blacklists

However, blacklists are not without their own issues. The impact of being blacklisted can be significant for individuals, organizations, and businesses. Here are some potential consequences:

  • Blocked email communication: If an IP address is blacklisted, emails sent from that address may be blocked or marked as spam by email providers, which can risk losing important mail.
  • Difficulty accessing certain services: If an IP address is blacklisted, it may be blocked from accessing certain websites or services.
  • Cybersecurity solutions use blacklist IP information to determine IP reputation. If your IP address is blacklisted, it means that the security solutions will block it. This means that it will affect direct mail and communication between organizations.

Furthermore, public addresses are frequently used as Network Address Translations (NATs), where the abnormal behaviour of one user can adversely impact numerous individuals. In our scenario, there are 100 users behind a single public address.

Research

During the research stage of AbuseBox’s development, we checked IP addresses blocked in the seven days prior to 28 June 2023 via AbuseIPDB. Mongolia ranked 54th in globally reported blacklisted IP addresses. During that week, 7,535 addresses were reported to AbuseIPDB, comprising 3.7% of the total. This figure was concerning, and we decided that action was needed.

Figure 1 — IP addresses blocked in the seven days prior to 28 June 2023, by economy.
Figure 1 — IP addresses blocked in the seven days prior to 28 June 2023, by economy.

We can determine the reasons for addresses reported to AbuseIPDB. Of the 25,244 addresses we checked, the top reasons are broken down in Figure 2.

Figure 2 — Most common block reasons for IPs reported to AbuseIPDB.
Figure 2 — Most common block reasons for IPs reported to AbuseIPDB.

For Mongolia specifically, we found nearly 40% of total IP addresses were blacklisted on 31 August 2023. As Figure 3 shows, we checked 156,976 addresses from the top 10 ASNs and 38.39% of these addresses were blacklisted. Figure 4 shows the sensors blacklisting Mongolian IP addresses.

Figure 3 — 38.39% of the top 10 Mongolian ASNs were blacklisted.
Figure 3 — 38.39% of the top 10 Mongolian ASNs were blacklisted.
Figure 4 — The sensors blacklisting Mongolian IP addresses.
Figure 4 — The sensors blacklisting Mongolian IP addresses.

Unitel’s solution

AbuseBox is our solution to regularly monitor IP addresses, return time-based reports, and automate delist requests to help ease the above risks from incorrect blacklisting. You can download AbuseBox from the GitHub repository

Using AbuseBox, if one of your IP assets is found to be blacklisted, you can send a delist request immediately. Currently, however, it is only possible to send a delist request to a few sensors. In principle, it is technically feasible to send delist requests for most sensors and this is currently being worked on.

To check if an IP is blacklisted, a method known as ‘DNSBL’ is used, and information is currently available from 48 different sensors:

def check_dnsbl(reversed_ip: str, provider: str) -> Tuple[bool, str]: 
  query = f'{reversed_ip}.{provider}'
  try: 
    response = socket.gethostbyname(query)
    return True, response
  except: # if name or service not know it means ip is not blacklisted 
    return False, ''

Suppose the IP address 1.2.3.4 is listed in the blacklist database of a provider. To verify this, a socket connection is made to 4.3.2.1, which serves as the DNSBL provider. If a response is received, it means the IP address is indeed blacklisted. Conversely, if no response is received, it confirms that the IP address is not blacklisted.

View the full source code for DNSBL on GitHub.

Web application architecture

To make AbuseBox we used Django (REST framework) for the backend and PostgreSQL as the database (Figure 1). VITE + React js is used for the frontend stack (Figure 2).

Figure 5 — Backend architecture.
Figure 5 — Backend architecture.
Figure 6 — Frontend stack.
Figure 6 — Frontend stack.

Using AbuseBox

Using AbuseBox is simple. To obtain reports and use other features you must first log in, where you’ll be met with the ‘user panel’, shown in Figure 3.

Figure 7 — AbuseBox user panel.
Figure 7 — AbuseBox user panel.

Next, you can add a hostname (IP address or domain name) that you want to monitor and submit (Figure 8). This will add the host to the continuous monitor list (Figure 9).

Figure 8 — Adding IP addresses to monitor.
Figure 8 — Adding IP addresses to monitor.
Figure 9 — Continuously monitored list.
Figure 9 — Continuously monitored list.

A report example is shown in Figure 10. You can initiate a delist request directly from this panel.

Figure 10 — Delist requests can be made directly from the report panel.
Figure 10 — Delist requests can be made directly from the report panel.

Delisting

Manual delist can be solved in just one click. Firstly, we require a proxy to capture web requests. Then, we convert the captured request into code, utilizing the Python request function (Figure 7).

Figure 11 — A captured delist request.
Figure 11 — A captured delist request.

We’ve developed a delist API for a select few sensors; the source code is available on GitHub. If there’s no authentication process on the sensor side, you’ll need to convert the captured web traffic into code that sends a delist request:

import requests

url = "https://www.blocklist.de/en/delist.html"

headers = {
    "Host": "www.blocklist.de",
    "Cache-Control": "max-age=0",
    "Sec-Ch-Ua": '"Not=A?Brand";v="99", "Chromium";v="118"',
    "Sec-Ch-Ua-Mobile": "?0",
    "Sec-Ch-Ua-Platform": '"Windows"',
    "Upgrade-Insecure-Requests": "1",
    "Origin": "https://www.blocklist.de",
    "Content-Type": "application/x-www-form-urlencoded",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.5993.90 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
    "Sec-Fetch-Site": "same-origin",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-User": "?1",
    "Sec-Fetch-Dest": "document",
    "Referer": "https://www.blocklist.de/en/delist.html?ip=",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.9",
}
	
payload = {
    "ip": "1.1.1.1",
    "action": "search",
    "g-recaptcha-response": "",
    "send": "delete",
}

response = requests.post(url, headers=headers, data=payload)

print(f"Response Status Code: {response.status_code}")

Potential risks of delisting

While we’ve devised a method for automatically requesting delisting from blacklists, it’s crucial to identify and correct the underlying reason behind the IP address being blacklisted. Our approach involves sending web requests, but ultimately, it’s up to the sensor to determine whether to remove the IP address from the blacklist.

Before sending a delist request it’s important to remember that if you don’t fix the reason for the blacklist, you will be blacklisted again in a short time. Varying amounts of time may be needed for delist requests, so it’s best to avoid repeated requests.

You can download AbuseBox from GitHub. The delist API for select sensor source code is also available on GitHub.

Watch Dashzeveg present this topic in detail during the Operations 2 session at APRICOT 2024.
Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

Leave a Reply

Your email address will not be published. Required fields are marked *

Top