Spamhaus still parties like it's 1999
I recently had visibility of a Spamhaus Block List (SBL) listing notification on the basis of malware being detected within a file delivered via HTTP/HTTPS.
As part of the report, they provide the affected URL (for the sake of this post we'll say it's https://foo.example.com/app.exe) along with details of the investigation they've done.
Ultimately that investigation is done in order to boil back to a set of IPs to add to their list.
Concerningly, this is, literally just
dig +short foo.example.comWhich gives them output of the form
CNAME1 CNAME2 1.2.3.4 4.5.6.7
They then run a reverse lookup (using nslookup) on those IP addresses in order to identify the ISP. The IPs are added to the SBL, and a notification sent to the associated ISP.
In this case, the URL was a legitimate file, though it had been bundled with some software falling under the Possibly Unwanted Application (PUA) category. The point of this post, though, is not to argue about whether it should have been considered worthy of addition.
The issue is that Spamhaus' investigation techniques seem to be stuck in the last century, causing potentially massive collateral damage whilst failing to actually protect against the very file that triggered the listing in the first place.
In case you're wondering why Spamhaus are looking for malware delivery over HTTP/HTTPS, it's because the SBL has URI blocking functionality - when a spam filter (like SpamAssasin) detects a URL in a mail, it can check whether the hosting domain resolves back to an IP in the SBL, and mark as spam if it does (in effect limiting the ability to spread malware via links in email - undoubtedly a nice idea).
Just to note, although they make it difficult to identify how to contact them about this kind of thing, I have attempted to contact Spamhaus about this (also tried via Twitter too).
It also seems only fair (to Spamhaus) to note that I also saw a Netcraft incident related to the same file, and they don't even provide the investigative steps they followed. So not only might Netcraft be falling for the same traps, but there's a lack of transparency preventing issue from being found and highlighted.
Underblocking
The whole notification came about because Spamhaus decided that https://foo.example.com/app.exe was malware, and they should protect their users against it.
As shown above, In order to collect the information required to do this they performed a single DNS lookup from a single location.
The problem is, the file is served via a CDN - DNS lookups from different locations (even within the US, where they tested from) will result in different results. So Spamhaus' investigation has, at best, protected consumers of their list in just one specific geographic location.
In reality (and in this case), it can actually prove to be less effective than even that though.
The domain doesn't point straight to a CDN, but first to a CDN switcher (similar to, but not, Cedexis' product). That switcher then apportions a share of traffic to each CDN the customer uses.
In our original example, the CNAME chain would look something like this:
foo.example.com 3600 IN CNAME customer1234.cdnswitcher.com customer1234.cdnswitcher 30 IN CNAME routedname.cdn1.com routedname 30 IN A 1.2.3.4 routedname 30 IN A 4.5.6.7
Note the short TTL's in the chain after the original CNAME out to the CDN switcher. A query 30 seconds later for that name might result in being directed to a completely different CDN:
foo.example.com 3600 IN CNAME customer1234.cdnswitcher.com customer1234.cdnswitcher 30 IN CNAME abcdef.someothercdn.org abcdef.someothercdn.org 30 IN A 7.8.9.0 abcdef.someothercdn.org 30 IN A 14.15.16.17
Essentially this means that an unknown (to the user) percentage of lookups will result in users going to a completely different CDN.
It's also complete pot-luck which CDN Spamhaus detects and blacklists during it's investigative stage: different queries a few minutes apart might result in them blocking Akamai instead of Edgecast (for example).
This won't have been visible to Spamhaus though - they passed +short to dig:
dig +short foo.example.comcustomer1234.cdnswitcher.com
abcdef.someothercdn.org
1.2.3.4
4.5.6.7
What they've done there is to, rather questionably (IMO), pass an argument who's specific intention is to remove information from the output.
Maybe I'm being overly harsh here, but it seems odd that anyone would do this whilst doing investigative work, it's essentially hiding relevant evidence from yourself, and dig's output really isn't that much longer without +short.
The chain of CNAMEs, though, should still have acted as a fairly major clue that this required much closer scrutiny than it got.
Knowing what we now know, we can conclude that even users in the geographic location that Spamhaus tested from aren't actually fully protected.
A substantial proportion of mails referring to this file might actually make it through without being marked as spam - it only needs the name to resolve back to another CDN, or a different PoP within the same CDN.
Where the SBL is used as an input to DNS for web-level filtering, then waiting for the connection to fail and pressing refresh might be sufficient to grant access (though this isn't, as far as I can tell, a configuration that Spamhaus support/recommend).
Useless Identifier of Future Sources of Spam
Now, readers may perhaps think that I'm being a little unfair to Spamhaus here - after all, the primary intended purpose of their product is to vet mail, and not as part of a generic web level blocking solution. Perhaps it's about preemptively identifying future sources of spam?
However, if we accept that, Spamhaus would still be wrong. Though I acknowledge this section may very well be a straw man, I felt it was worth addressing an obvious possible argument.
Even if we could take presence of malware as conclusive evidence that a site has been compromised, there's still a major issue in the approach: CDNs do not host applications. They're glorified caching reverse proxies.
Even if there is some truth in the idea that a site hosting malware has probably been compromised and so may eventually start sending spam, it's the origin - the server hosting the application - that'll be generating that spam, and not the edge nodes.
The origin sits behind the CDN and isn't included in the DNS results:
As a result, the very machine which might send spam in the future is the one who's IP will not have been included in the SBL, whilst a bunch of nodes which probably aren't even running a MTA will have been.
So to bring this section to a close, we can summarise as follows:
- if the intention is web-filtering, then the SBL is significantly underblocking the targeted content.
- If the intention is to identify and proactively block a possible future source of spam, then the SBL is failing to block the correct IPs at all.
- An unknown percentage of mails in the tested location would get past spam filters using the SBL
- All mail in other locations would get past spam filters using the SBL.
In all these cases, for the vast majority of their users, the SBL is offering little more than a false sense of security. Given the level of CDN usage on today's internet, the example I've seen simply cannot be an isolated instance.
Overblocking
I mentioned above that the file in question is served via CDN.
Most CDNs serve more than one domain/customer, and all of those customers are affected by inclusion of a node's IP in the SBL, where it's used for web level blocking.
Where the SBL is used in web-level blocking, the result will be silent failures for some users, as the network infra they're on either refuses to return DNS resultsets containing those IPs, or (perhaps more commonly) their edges defences drop packets destined for these nodes.
However, the damage is actually wider than that. The SBL is designed to integrate with spam blockers that implement SBL_URI:
Yes, the SBL can also be used as a URI Blocklist and is particularly effective in this role. In tests, over 60% of spam was found to contain URIs (links to web sites) whose webserver IPs were listed on the SBL. SpamAssassin, for example, includes a feature called URIBL_SBL for this purpose. The technique involves resolving the URI's domain to and IP address and checking that against the SBL zone.
CDNs don't commonly serve just one class of customer, so the breadth of collateral damage can be pretty significant. Imagine finding that your favourite music service suddenly doesn't work. Or indeed, that you're trying to troubleshoot that but some of the online resources are themselves blocked.
Where your mail spam filters use SBL_URI you may well find that legitimate messages end up in spam because they use the same CDN.
Of course, it's not just CDN's which are affected by this, it could equally well be shared hosting.
The root cause of this is that Spamhaus' approach seems to be stuck in the last century: when HTTP/1.0 was king, there was no Host header so 1 IP meant 1 website. That's not been the case for well over a decade - even the relative latecomer in that respect, HTTPS, got Server Name Indication (SNI) to allow multiple certificates per IP back in 2003.
Conclusion
Spamhaus' lists exist to protect businesses and users alike. But, because of failures in their investigative technique the result can easily fall somewhere between overblocking and a false sense of security, with no way to reliably predict which it'll be for any given user.
Their FAQ's state that avoiding false positives is a priority
The SBL's primary objective is to avoid 'false positives' while blocking as much spam as possible. Indeed because SBL false positives are extremely rare, there is little visible controversy regarding the SBL yet we are one of the Internet's biggest spam blocking systems.
The problem with IP level blocking, as it pertains to HTTP delivery, is that there is insufficient granularity to effectively avoid false positives, which renders it more or less incompatible with their primary aim (at least as far as SBL_URI is concerned).
If IP level blocking is going to be used, then during the investigative stage:
- lookups should be performed from multiple locations
- multiple lookups should be performed in in each of those locations
- careful examination of the full resultset should be performed (i.e. no more
+short, things like TTLs are often an important indicator).
If delivery is via CDN then collateral damage is almost certain from both web level blocking and SBL_URI - The majority of links resolving back to those IP's won't be spam/malware related but will get treated as such.
Handle CDN's specially
Where investigation shows a significant likelihood that a CDN is involved then that CDN provider should be contacted (details will generally be available by running whois against one of the earlier CNAMEs), with subsequent addition to the SBLs if the provider isn't responsive.
CDNs after all, are teminators at both TCP and application level, so are fully capable of blocking access to a URL being served via their infrastructure - giving exactly the level of granularity required to help avoid overblocking.
IP level blocking is harmful at web-level
Although there are exceptions, IP level blocking is far too blunt a tool for use at web level, and no-one should really be using the SBL as a final decision for filtering Web access (I grudgingly include SBL_URI here).
Even in the adblocking world, it's been found that domain level blocking isn't sufficiently granular, despite being better than IP level blocks. Full URI level filtering is required along side to catch the many instances where blocking the underlying domain causes unwanted side effects.
Malware may be more serious than advertising, but the underlying principle remains the same, and that need for granularity is why things like Safebrowsing came about (despite some recent controversy surrounding Apple's implementation).
Maintain the integrity of an important system
Spamhaus' system works (correctly IMO) on the principle that the receiver/DNSBL user should be able to make an informed decision about a given IP, and that Spamhaus' DB just offers an "opinion" based on the information it holds. But, because Spamhaus' initial investigation collects incomplete information, DNSBL users are ultimately (unknowingly) making decisions based on incorrect or incomplete information, and opportunities are being missed to add the correct information.
Addition to Spamhaus' lists is (and in my opinion, should remain) considered a serious event - the lists are a valuable tool.
That's severely undermined though when the quality of the information being fed into their lists becomes questionable
No-one expects the lists to achieve 100% coverage, but it probably is reasonable to expect that sufficient care has been taken to avoid overblocking (Spamhaus seem to expect it of themselves too) as well as to avoid listing the wrong IPs in the first place, and in particular that the investigative approach used is applicable to the modern world.
Perhaps I'm being unfair, but I don't feel that Spamhaus' investigative approach currently lives up to the burden of responsibility which comes with maintaining such an important tool, particularly given the inherent lack of granularity in any implementation using SBL_URI.
For avoidance of doubt, my objection in all this wasn't Spamhaus' concern about the URL, or even their inclusion of information in the SBL.
My objection stems from the fact that their approach resulted in wrong, incomplete and irrelevant information being added to the SBL - if that's something they're doing regularly then it significantly reduces the value of the SBL as an indicator in the fight against spam and malware. Although it's probably harmless (if ineffective) when used to vet mail senders, the existence of SBL_URI means that mistakes can cause significant collateral damage.
