DNS Woes

While much of this blog will be of use to AH employees – I, personally, have greatly benefited from reading tips and tricks from various blogs around the Internet. Sometimes people have the exact same problems I do, and I get to benefit from their solution!

I’d like to make sure some of my solutions can be of benefit to others, so I’ve created this ‘technobabble’ category. Feel free to ignore unless you’re really, really interested in the nitty-gritty of AHIT.

We recently changed ISPs across a number of sites. While I had been testing them at a couple of our sites over the course of the year, I didn’t check up well before rolling out this larger change. As a result, we found that this ISP had started using a transparent proxy. Unfortunately this meant that OpenDNS stopped functioning, leaving lots and lots of undesirable things accessible from our campuses.

We successfully mounted an application to be removed from the transparent proxy. Under normal circumstances, this would solve our problems – but it looks like beyond just the proxy, DNS requests are being rerouted.

Not, however, in a opaque and easy to detect way. It seems as though common DNS servers are being rerouted through a transparent DNS proxy… but requests that would fail otherwise, still fail.

The upshot is I’ve created an offsite DNS server that simply grabs results from OpenDNS directly through another ISP. With caching enabled on-site, I suspect that performance will be acceptable. The upside to this approach is that, once you’ve got BIND configured, it’s incredibly easy to deploy. No need to worry about Dynamic IPs or OpenDNS’ updater. You plug in your private DNS server and it ‘just works’.

The downside to this approach is that I won’t have the fine-grained control that OpenDNS provides in its native implementation. Sites are blocked everywhere, or nowhere. Worse still, I don’t have the reporting that lets me know when something nefarious is happening at a single site.

We’re still working with the ISP to figure out what’s going on, but this solution does solve our immediate need.

Some handy tools for diagnosing DNS problems:

  • namebench - benchmarks DNS servers, automatically detects redirects

  • DNS Leak Test - lets you see where your DNS queries are being resolved