NOTE: This version is outdated but remains for posterity; use the continually updated version for the best practices to follow.
As noted on several occasions over the years, malicious hackers have hijacked advertising networks for the purpose of spreading malware on even the most mainstream and seemingly innocuous sites, like the New York Times and Drudge Report; for this reason, ad-blocking is not merely a matter of making your browsing experience less cluttered and faster (because less content will be downloaded), it's also a matter of keeping yourself safe on the Internet. With the advent of content blocking ability in version 5 of both Google Chrome and Apple Safari, it is now possible to almost completely seal off this security hole regardless of which browser or desktop operating system you use. Following is a description of ad-blocking followed by some techniques to block ads, some of which are like Ronco ("Set It, and Forget It!") while others must be updated manually but do not need to be updated often, possibly once per month.
- Basic yet Non-Essential Information
- Internet Explorer on Windows
- Firefox and Related Browsers, Even Mobile
- Chrome, Chromium, and Derivatives
- Safari 5+, for Windows or Mac
- Opera: Yes It Can Be Even More Secure, Even When Mobile
- The Rest of the Lot
- Ad-Blocking for Everyone
- Automate Your Protection
- Ad-Blocking Isn't Everything
Whether built-in or available via an extension, all of the major desktop Web browsers (and some mobile browsers) allow for lists of URL patterns to be maintained that are then used to determine whether to block content, and in some cases to override the blocklist with a whitelist to allow particular sites to load certain content anyway; some allow for these lists to be automatically updated, and others must be updated manually, but it is not a particularly difficult task. The cornerstone of protection from malvertising is ad-blocking (which may be described as "filtering" of "content" by the vendor to maintain a good corporate image).
Often advertising is intended to load into a block-level HTML element, like a
iframe, and when it is blocked, a noticeable blank space remains; some ad-blocking systems have element hiding rules built-in to collapse these spaces, but at least one requires a special CSS file for this purpose.
The HOSTS File
From the early days of ARPANET (the predecessor to the Internet, founded by the United States DARPA), computers would locate each other via numerical addresses; these were difficult to remember, so a system administrator at SRI would maintain a text file called HOSTS.TXT that all computers on the Internet would retrieve showing the relationship between an address and one or more hosts, given by their domain names, because researchers could more easily remember mit.edu than 22.214.171.124 or whatever address the main MIT Internet site (not "website" for this was long before the birth of the WWW in 1990) had on that particular day.
By 1983, the Internet, although not even a household name yet, had grown so large that the bandwidth required for all of the requests for HOSTS.TXT swamped SRI, so a more decentralized system for mapping domain names to what were then known as IP addresses was invented, DNS. By this time, the HOSTS file had been enshrined into the TCP/IP standard (the protocol stack underlying the Internet), and to this day, if your computer finds a domain name in its HOSTS file, that will pre-empt a DNS lookup.
All HOSTS files must contain the line
127.0.0.1 localhost near the beginning, followed by
::1 localhost if IPv6 is enabled, and
255.255.255.255 broadcasthost on Mac OS X, or else you may have difficulty accessing the Internet; the format is slightly different for the classic Mac OS, and if you are still interested in maintaining that obsolete operating system it is not hard to find out.
It is possible to use an alternative DNS provider like OpenDNS, or set up a filtering on your router, for yet another layer of protection, but this is more difficult to do and will not be covered here in detail; mostly this is useful only to keep users from reaching particularly popular websites, like Facebook, YouTube, MySpace, or Twitter (or many more sites, as a form of parental control), and although OpenDNS has value as a DNS provider that may be superior to your ISP's DNS, its filtering is a bit heavy-handed for those who merely want to be safe from malware while fully experiencing what the Web has to offer.
Some browsers have ways to prevent certain cookies from being set (thereby preventing advertisers or more nefarious types from tracking your behavior) or plugins or ActiveX controls (Internet Explorer's version of "plugins") from being loaded, or sites from running scripts or other active content.
The Layers of Protection
This sketch is not complete, and not all layers will be dealt with here, but it gives you an idea of where an HTTP or similar request can be stopped:
- Your browser's ad-blocking system may find a URL matching a pattern in its blocklist but not in its whitelist.
- If that fails, your operating system may find the domain name in its HOSTS file, redirected to an invalid IP address like 0.0.0.0, and return nothing.
- You may be running a filtering proxy like Privoxy or the Proxomitron (not covered here), or an external link scanner (also not covered here), that drains resources but can alter a webpage in any way it wants, including keeping that URL from going through.
- If that fails, your operating system's firewall, your router, or your DNS provider may keep you from reaching the domain name specified.
- If that fails, it is still possible for your request to be blocked by the server (but this usually means that your IP address has been banned from that site, and this is almost never desirable and never under your control).
- Finally, if the request makes it through, whatever the server sends out to you may still be prevented from doing much, if the browser has been immunized against letting the site run scripts, set cookies, or load plugins; some plugins or ActiveX controls may be prevented from running at all.
Protecting Internet Explorer on Windows
InPrivate Filtering: IE8 and Above
The most exciting innovation in content blocking was when in March 2009, Internet Explorer 8 was released with support for InPrivate Filtering; it took a while for people to realize its power, however, because by default InPrivate browsing is not turned on.
You can find detailed instructions here about a registry entry that will make Internet Explorer launch in InPrivate mode by default and how to download an updated XML file containing a list of URLs to filter, based on a list by Fanboy, who makes filter lists for Opera, SRWare Iron, and the many "AdBlock" extensions in other browsers; the most important thing is to delete your InPrivate Filtering entries before importing the new ones from the latest version of this XML file, which I try to mirror here (as "adblock.xml"). Access your InPrivate Filtering by entering Internet Options (either from within IE or from the Control Panel), clicking the Programs tab, and choosing "Manage Add-Ons"; if you don't see "InPrivate Filtering" that means you aren't using IE8, and then if you are using Windows XP or later, get the latest version of IE (8, or soon 9 if using Vista or later) right away.
If you read further, into what you can do with the other browsers, you may read about a certain user stylesheet for element hiding; it does not work in Internet Explorer, which lacks the sophisticated understanding of CSS necessary to keep from completely hiding a whole page rather than an advertisement on it.
The Restricted Sites Zone
The Restricted Sites zone (available since IE4) is a list of domains for which especially stringent security policies are enforced; among other things, Restricted Sites may not run scripts, Java applets, or ActiveX controls. However, these sites are not actually blocked, the usual methods of automated immunization use the same set of domains for both the Restricted Sites zone and the HOSTS file (thereby obviating the placement of the domains into Restricted Sites, as they cannot be accessed anyway), and especially shortly after the launch of IE8, Internet Explorer has opened slowly whenever the Restricted Sites zone was large.
To remove sites from the Restricted Sites zone, download ZonedOut, select the "Local Machine" hive, be sure you're at the Restricted Sites zone, choose "Remove All" from "Remove Sites" in the Menu, go back to "Current User" and do it again, and then exit; then if you have used SpywareBlaster, open it, go to Restricted Sites, right-click inside the list to de-select all, then Remove Protection, and uncheck the box (the other immunizations are still useful, against certain ActiveX controls and tracking cookies) and exit. Finally, if you used Spybot to immunize IE, open it, right-click in the Immunize screen to de-select all, then select all the Internet Explorer entries related to "Domains" or "Secure Domains" (leave Cookies, IPs, Plugins, and anything other than IE alone) and press the Undo button; then de-select all, select all the immunizations you didn't just Undo, and close Spybot.
If you would actually like to add more sites to Restricted Sites in an automated manner, I have some text files from the IE-SPYAD project as well as a mega-file merging them and containing the additions to Restricted Sites from Spybot and SpywareBlaster in my
/etc/risky subdirectory. As with InPrivate Filtering, you can add individual entries to the Restricted Sites zone from Internet Options; this time go to the Security tab, click the red symbol for Restricted Sites, and press the Sites button (you can also change security settings for any of the zones here).
It is possible to use SpywareBlaster and Spybot to keep some tracking cookies from being set or ActiveX controls from being loaded, and often Microsoft updates will include more ActiveX killbits; also if you're up to the challenge and tedium, you can set the security of the Internet zone to keep scripts and ActiveX controls and the like from running, and then you can put the few sites you trust (like Microsoft Update) into Trusted Sites, but it's a pain to maintain it because whenever you have a new site to add you have to go back into Internet Options...
The principal method is via the AdBlock Plus extension, for Firefox and derivatives, including the mobile version; the EasyList subscription is most recommended, and you may wish to subscribe to the related EasyPrivacy list, a list of Malware Domains, Fanboy's adult/dating ad list, and even a list to keep you from being rickrolled (but don't add too many lists or your memory usage will spike); AdBlock Plus has an expressive syntax that allows very specific URL patterns to be caught and whitelisting and element hiding like no other system, and it is so successful that it has been copied by other open-source browsers like Arora, Konqueror, and to a lesser extent QTWeb. It is even possible to right-click on any page to block individual elements, although finer-grained blocking, like blocking ads inside YouTube videos, is best done by the authors of the filter lists.
You may also be interested in the Element Hiding Helper if you want to help create element-hiding rules, which collapse the spaces occupied by advertising; also this CSS file may be of some use (be sure to rename it
userContent.css and place it in the chrome directory of your user profile), but it's probably not needed.
It is possible to use SpywareBlaster and Spybot to keep some tracking cookies from being set or popups from launching, but more important is the NoScript extension, which keeps all scripts and plugin content from loading by default; it takes a while to set scripting permissions to just the sites you want, but it's much easier than the Internet Explorer way because you can just click the icon in the taskbar to see all the domains that want to run scripts (you probably want the scripts from the main site to run, but not advertisers or analytics scripts). Also, although most of this advice does not apply to the stripped-down Firefox derivative Camino, for Mac OS X, it does come with a built-in plugin blocker for Flash, Shockwave, and Silverlight, which works similarly to NoScript in this area.
Similarly to there is an AdBlock extension that allows you to subscribe to filter lists and is starting to be able to block content (due to an update in the underlying WebKit engine that also powers Safari), but it doesn't do it quite as well so I prefer the system developed for the Chromium derivative SRWare Iron back when Chrome lacked extensions, let alone a working ad-blocker; Iron uses a file called adblock.ini in its program directory, and although it does not auto-update, cannot be updated while Iron is running, and features a limited syntax, it does an excellent job at blocking content. Pre-made files are available from Fanboy, and I have occasionally combined them with Spybot's immunization for Opera to make a comprehensive adblock.ini file
Whether using Chrome, Iron, the Flock 3 Beta, or any other derivative of Chromium, the Adblock+ Element Hiding Helper may be of interest; although AdBlock recently gained the ability to collapse blocked content, this extension also does the trick, which is nice because apparently these browsers do not support user stylesheets.
The only form of immunization I know about against auto-loading content is the FlashBlock extension, which blocks the most-commonly used plugins and lets you start them when you feel like it, but otherwise is more limited than NoScript for Firefox.
Protecting Safari 5+
As soon as Safari 5 was released, with support for extensions, the developer of AdBlock for Chrome ported it to Safari, and it works even better at keeping content from loading; the current version has no support for element hiding so instead this CSS file, saved somewhere and used as a user stylesheet in Safari, may be used instead.
The Plugin Blocker extension very nicely prevents plugins from loading automatically, although it doesn't keep scripts from auto-loading like NoScript; for older versions of Safari (for Tiger and earlier versions of Mac OS X with which Safari 5 is incompatible) there is ClickToFlash.
Ever since Opera 9 in 2006, including Opera Mobile and I believe Opera Mini, it has been possible to block content by right-clicking anywhere on a page and selecting images to block; the URLs to block were stored in a file called urlfilter.ini, which was originally designed for a kiosk mode, in which users could be forced to navigate to only certain sites. The file can also be used to more systematically block content; Fanboy has been maintaining URL Filter files for Opera, and Spybot has been adding entries to this file as part of its Immunization routine. Its syntax is powerful, but not quite as much as AdBlock, and there are no whitelists, but in practice that is not a major issue; also there is no element hiding, so Fanboy has also made this CSS file to be used as a user stylesheet. If you would prefer a massive compilation of all of the regional lists and Spybot's immunization (which may be helpful because since Opera 10 rearranged things, Spybot has had a hard time detecting urlfilter.ini), I have provided it, along with a plugin-ignore.ini file as mentioned below.
Spybot also adds entries to a file called plugin-ignore.ini to keep plugins from being loaded and to a file called cookies4.dat to keep certain cookies from being set, but the latter is imperfect because Spybot wipes out the existing cookies, and as soon as Opera is used again and new logins are made, Spybot thinks Opera's cookies aren't immunized anymore; also the plugin-ignore.ini file may cause a lot of plugins to fail to be loaded, for it seems as if once a plugin is found in the plugin path or enumerated from the Registry that matches this file, Opera doesn't look for any more plugins, and you may come to think that QuickTime or your PDF reader is missing.
There is a good chance that your browser implements some variant of AdBlock and allows for a special element-hiding user stylesheet, and even if you don't know a way to block content from within the browser you may still be assisted with...
Protection via the HOSTS File
As mentioned earlier, the HOSTS file takes precedence over the usual DNS lookup; this can be used to our advantage, by redirecting certain domain names to other IP addresses (some malware uses this same trick against us, but it's easy to remedy). The usual redirection addresses are 0.0.0.0 (short, known to be invalid) or 127.0.0.1 (the local computer...unwise if you're running a web server), and if you ever want to undo the effect of an ad-blocking HOSTS file, replace its contents with the line(s) mentioned in the explanation of the HOSTS file; on Unix-like systems like Linux and Mac OS X, it is found at /etc/ while on Windows it is found at %SystemRoot%\System32\drivers\etc\ and has the name "hosts" with no extension. The advantage of protection via the HOSTS file is that is works on anything that uses the Internet, but the disadvantage is that it works per domain, with no wildcards allowed; either an entire domain is blocked or no URL on the domain is blocked, so a site needs to be wholly malicious or annoying before it ought to be added to an ad-blocking HOSTS file.
For most purposes, the HOSTS file available from Peter Lowe should suffice; just download it (use this format for classic Mac OS), save it as
hosts with no extension, and replace your current HOSTS file. On Mac OS X, you may wish to then use the command
sudo niload -v -m hosts . < /etc/hosts in a Terminal window to update the NetInfo database, while in Windows you should stop the DNS Client service and set it to Manual in the Services panel (Press WinKey+R, then type
services.msc into the Run dialog, then look for the DNS Client), because its DNS Cache does not actually speed up browsing but does bog down in the face of a HOSTS file more than a few dozen kilobytes in size. On Windows you can also supplement this file with Spybot's immunization, and there is a Registry tweak so that Spybot will use 0.0.0.0 instead of 127.0.0.1 as its redirection IP.
Automated HOSTS Management: Windows Only
If you want to do even more, download HostsMan and of the pre-set update sources, select only Peter Lowe's AdServers, for MVPs and especially hpHosts are too massive and contain too many false positives (like file-hosters MegaUpload, FileFactory, and RapidShare and the open-source software repository SourceForge); you may consider adding this list I made from SpywareBlaster's Restricted Sites list as an update source. You should consider changing all redirection IP addresses to 0.0.0.0 and eliminating comments and duplicates, and then making a backup just before optimization (which makes use of the fact that each line may associate several domain names with one IP address); the purpose of the backup is to be swapped out for the next time you re-immunize with Spybot, otherwise it will think more than 10,000 domains have been left unprotected.
Of course if you would rather just enjoy the fruits of my own labor, I am willing to provide it; at just over a third of a megabyte, this HOSTS file is rather large but not so much that it slows browsing unacceptably, and it is as small as possible for the number of domains it blocks.
Free Programs for Automating Protection: Windows Only
Spybot - Search & Destroy was originally famous as a spyware scanner, but it has fallen behind in both speed and comprehensiveness and is now only good for its immunization ability; it is recommended to keep the resident TeaTimer and the SDHelper IE plugin from running, and to keep Spybot from immunizing anything about IE except for cookies and plugins, but it is excellent at bolstering the security of Opera, adding entries to the HOSTS file, and keeping some ActiveX controls from harming your computer. Its database updates on Wednesday mornings.
SpywareBlaster from BrightFort adds more ActiveX killbits and keeps certain cookies from being set in Internet Explorer or Firefox; its use of Restricted Sites is no longer recommended, however, and like a properly-setup Spybot, it won't use system resources unless you launch it and immunize, and it doesn't run in the background when you close it, unlike a resident anti-malware program.
HostsMan allows you to automatically update and optimize your HOSTS file, change the redirection IP (the invalid 0.0.0.0 or the local computer 127.0.0.1), and even run a small Web server that delivers nothing to any request for 127.0.0.1 on port 80 (HTTP, or unencrypted Web traffic); this is best used right after re-immunizing with Spybot.
A Final Note
Do not be lulled into complacency: It is still possible for a sufficiently determined cyber-criminal to directly hack into the websites you visit, not just their advertisers, so you must be vigilant. To eliminate vulnerability to ActiveX exploits like in the article I just linked to, use any browser other than Internet Explorer except when you must (like using Microsoft Update in versions of Windows before Vista, and even then IE Tab Plus for Firefox or IE Tab for Chrome work wonders), and more generally ensure your operating system, browsers, and especially plugins are up-to-date (because many vulnerabilities arise in Java, Flash, and especially your PDF reader…and no, it's not just Adobe); Mozilla has provided an excellent service for almost any browser (it works best in Firefox 3.6 and later, almost as well in most browsers, and well enough in Internet Explorer) called Plugin Check to check on your plugins, and remember that Flash for IE and Flash for all other browsers are different. Remember that you aren't safe just because you don't use Windows. Also be sure that your firewall and router (you should get one, rather than directly connecting to your modem) are set up to keep port- and IP-scanners from knowing you even have any ports open (unless you are running a Web server or something similar), by checking with GRC's Shields UP!