Some time ago someone reported issues accessing cpan.catalyst.net.nz a little peculiar and puzzling at the time, but we put it down to some weird DNS cache issues and moved on.

Turns out the problem is a DNS one, though not what we were thinking at the time:

$ host -t aaaa mail.catalyst.net.nz
mail.catalyst.net.nz has IPv6 address 2404:130:0:10::40:0

2404:130:0:10::40:0 == 2404:0130:0000:0010:0000:0000:0040:0000

$ echo $((0x24)).$((0x04)).$((0x01)).$((0x30))
36.4.1.48

So somewhere, some crappy device is getting a bunch of bytes back when it asks DNS for the address of something, and then it's taking the first four of them and calling that the IP address.

Kudos to David Clarke for spotting the actual problem.

I found another interesting avenue for affecting a web application recently when Heather was trying to renew one of her magazine subscriptions. She mentioned that the site was getting a '500 Server Error' and I recognised the e-mail address it was suggesting, so I banged an e-mail off to advise the problem.

Curiously, they weren't able to duplicate the issue while I was still seeing the problem. I did a little fooling around and discovered that I only saw the error when I was making the request through my proxy server.

A little more digging and I ascertained that if I connected to the proxy normally via IPv6 I got the '500 Server Error', but if I instead connected to the proxy via IPv4 it all worked just fine.

I've been helping Amos Jeffries with a little testing in the last week to help nail some IPv6 bugginess preparatory to the upcoming 3.1 release of squid. In the process of that I've built some Squid3 packages with IPv6 support enabled from current HEAD.

Get 'em while they're hot.

Note that these are in a 'works for me' state. They have been built on Lenny, and I have them running on both Lenny and Sid. I haven't put them somewhere you could apt-get them from because you should be paying attention if you're going to use them!

PS. If you can't click through to Amos' site it's because you're using IPv6 and the EveryDNS servers are continuing to serve up old data for his domain. Sigh.

The IPv6 wave progresses apace. Well, perhaps not 'apace', but it is moving...

The latest kernel exploit has incidentally had some local fallout in causing more of our boxen here to be upgraded to kernels with IPv6 support, and as a consequence our mail server is now reachable on IPv6. Some have suggested that making it only reachable on IPv6 is a good solution to spam but I suspect that there are still a few mailservers out there that we do want to receive e-mail from which are not IPv6 capable yet!

Since I can now SMTP and IMAP happily over IPv6 I decided it was time to get more adventurous. IPv6 is now in Squid3 head, so I built Etch packages of that and it seems to be 'basically working' in a few places now. We've been using ircd-ircu for a long time for an IRC daemon and it similarly seems that now has IPv6 support, so I backported that to Etch as well.

Packages are available for i386 and amd64 from my repository:

deb http://debian.mcmillan.net.nz/debian etch ipv6
deb-src http://debian.mcmillan.net.nz/debian etch ipv6

If I think of more Etch things that I need for IPv6 I'll put them there too. I do have dircproxy for Etch with support for connecting to IPv6 ircd but I seem to have misplaced the packages somewhere. If you're keen on seeing that then I'm sure I can reconstruct them somehow...

Now that we are having increasing amounts of IPv6 around some things are starting to reduce down to a 'Small Matter of Firewalling', which is suggesting to me that we will need manage our firewall rulesets differently for IPv6 than we have for IPv4.

In a lot of cases we can turn on/off large chunks of access related to a particular person/organisation by disabling a VPN, with the firewalling being a somewhat static monolithic overriding control above that. With the control potentially moving away from the VPN, and more directly into the firewall rules, we will need clearer association mechanisms in place. Of course we will continue to have VPNs, but they might become somewhat simpler, reducing in many cases to encrypted tunnels between exact endpoints.

The availability of IPv6 worldwide is surprisingly extensive, nowadays, but over the past years as it has slowly filtered around, people have had bad experiences with it because of poor routing. It seems that, as always, a bad rep travels about 20 times further than a good one, so an automatic response to casual problems that people see when using IPv6 is to blacklist it, without actually investigating the problem.

Take today, for an example. Someone said to me "I disabled IPv6 in Firefox because it was slow for one of my favourite sites". OK, so show me this favourite site. Show me the traceroute. Give me some facts!

Further investigation showed that although there is an AAAA record for www.crooksandliars.com, there is nothing listening on the other end! Looking at the AAAA record returned, we see that it is an autoconfigured IPv6 address of the form xxxx:xxxx:xxxx:xxxx:xxxx:xxFF:FExx:xxxx and can conjecture that the likely problem is that someone has done a hardware upgrade on their server, so they now have a different MAC, and consequently the autoconfigured IPv6 address has changed. Other scenarios are entirely possible, of course, but this is a likely one.

This is the second case I have seen where someone was running publicly available services on IPv6 using a manual DNS record pointing at an automatic IPv6 address, but I doubt that it will be the last. Unless there is infrastructure in place to automatically update your DNS when your address is autoconfigured, you are going to get bitten by this problem at some point if people remotely connect to your system for some service.

A recent thread which started on the Debian Release mailing list caught my eye this week. I attempted to aid the migration of this thread to the debian-ipv6 mailing list, which is really a better place for this and sorely in need of controversial topics for discussion.

It is interesting how people can so blindly decide that broken things should be destroyed. Repair often appears not to be an option, even for a long-term, wide-reaching effort like this, though we are all working on open-source software!

In this case there are an unknown number of less fortunate people in the world who are located behind some kinds of broken DNS infrastructure which discards 'AAAA' lookups. Of course 'AAAA' lookups are attempts to resolve a name to an IPv6 address, and the resolver in a 'modern' libc (i.e. one from the last five years or so :-) will try to retrieve an IPv6 address before it attempts to resolve a name as IPv4 with an 'A' lookup. That is how the standard is written, so if you want to comply with the standard you have to do it that way. Other things also interfere, but this element of the specified behaviour seems to cause the most annoying and pointless whingeing I have heard.

I suppose that the people who want working IPv6 make it so, and do not have problems with this behaviour. But it seems that people who are behind this kind of broken DNS either disable IPv6, or they have to whine about IPv6 being turned on by default, and can't we please all go back to the good old days. What's wrong with IPv4 anyway? Doesn't NAT solve all of it's problems? Are we sure this new (heh!) technology is safe?

Fortunately some people are so good at making their pain felt by other people that they can get other people to do their work for them. So Mithrandir has written a nice elegant patch for libc6 so that it won't do the IPv6 lookups unless you have a usable IPv6 configuration. I've filed bug #435646 against Debian to get this included, but Aurelian Jarno justifiably wants a few people to test it a bit harder... So I've taken the original patch and tweaked it to apply against current libc6 sources (2.6-5) and tested it for myself. It works as desired, as far as I can see, when comparing behaviour with an unpatched system. The patch is attached to the bug report, of course.

Perhaps some other people out there can put a wee bit of CPU into testing this for other environments so that we can make life easier for those people with no time / inclination to use IPv6, to ensure that they don't just disable it because it is making life too painful for them in it's current form?

How to build libc6 for fun and proft

If you have appropriate deb-src URLs configured, and are running Sid, then the following will let you build a local copy of libc6 with the patch. This is probably better testing than if I just make my packages available (which are only i386 in any case).


apt-get build-dep glibc
apt-get source glibc
cd glibc-2.6.1
debian/rules unpack
cd build-tree/glibc-2.6.1
wget -Oglibc-only-lookup-ipv6-if-it-makes-sense-debian.patch http://tinyurl.com/3xzm3o
patch -p1 <glibc-only-lookup-ipv6-if-it-makes-sense-debian.patch
cd -
dch --newversion 2.6.1-1+v6 "Apply IPv6 Resolver Sanity"
fakeroot debian/rules binary


Wait a few hours for it to build...

Install...

And then confirm that this only does 'AAAA' lookups if (when) you actually have a global or site scoped IPv6 address. When you only have a loopback or link local IPv6 address then you should only see 'A' record lookups.

Step 3: profit!

Whoops! I forgot the fun bit: please update the bug report :-)

What the patch does

In my opinion this is quite an elegant solution from Tolleff. He has picked a single characteristic of the IPv6 interfaces to further refine whether the IPv6 configuration of some interface is actually usable.

With IPv6 it is much more common to have multiple addresses assigned to a single interface. Interfaces are automatically configured with a link-local address which is not globally routable, and the loopback interface is also configured with the IPv6 equivalent of 127.0.0.1 (which is "::1").

To get a usable IPv6 setup you will also end up with a more widely usable address. In most cases this will be a global address, meaning that it is (in theory) globally routable from other people who also have global addresses, or you could have a "site" address, which is the IPv6 equivalent of RFC1918 addressing.

The patch considers that for the purposes of name resolution, it will be pointless to do AAAA lookups unless you have an address of the second kind. This means that people behind broken DNS won't be impacted unless the try and set up IPv6, and people who don't try and set up IPv6 won't get the 'hesitation' while their system attempts to resolve each address in IPv6 space first.

It will also mean that when people start to enable IPv6 around them, their setup will continue to work correctly.

I'm at DebConf7 now, in Edinburgh, and it is great to catch up with the great bunch of geeks that is the Debian community once more. As well as working on general stuff through DebCamp (the week leading up to DebConf proper) I'm helping out in producing the video streams running the sound desk or one of the cameras. The streams are highly recommended, and you can find them on http://streams.video.debconf.org:8000/, though for the New Zealanders reading this they do, of course, happen at seriously silly times of the day. Here is the DebConf7 schedule if you want to watch :-(. We'll get the talks up in a downloadable format as soon as we can, and I'll point people at them again then.

Highlights of the conference so far have been a talk by Nick Mailer on Debian Day, which was a very nicely presented run-down on the ancient history of intellectual property laws here in the UK, and the very interactive session this evening on Maintaining Packages with Git, which had some very good tips from David Nusinow as well as a lot of support from the audience. I now better understand why Penny is so keen on "git rebase", though hopefully I would have figured that out if I worked on more shared projects.

It's great to see 150-odd people in the room, many of whom had some experience with Git, and there were some very heavy users with great insight into it's strengths and weaknesses. Contrast that with DebCon5 in Helsinki where I almost felt Martin Langhoff and I were unheard when we acclaimed it, or even at LCA in 2006, where Martin did a first pass at importing X.Org into Git. Of course Git itself has come a long way from then too.

Another very interesting session was Martin Krafft talking about 'netconf', which is his proposal for a replacement of the ifupdown infrastructure using a much more stateless, event driven approach. This definitely looks like a well considered design and I hope he can pull it off.

I've got my own talk on Thursday, so I'll be concentrating on pulling that together for the next few days. If as many people come as have said they will (50% of the people signed up for any talk at that time) then it will be a very stuffy BoF room. The venue here in Edinburgh is great, and it's nice to have a good network connection. Everything has been running very smoothly for the participants, as far as I can see, though I know plenty of people have been working very hard to make it so, behind the scenes.

Weirdly, after only a short time using IPv6 at home and at work, I am finding I miss not having it readily available IPv6. I unfortunately had to disable my tunnels because adding 350mS latency to everything makes the web very frustrating. Perhaps next year we'll be able to add IPv6 to the desired network features for DebConf8 in Mar del Plata. Good to see that in the meantime the availability of the new IPv6 nameservers for .nz has been announced in my absence though.

From today my blog is available on IPv6. I've set it up so that http://debiana.org/ is only serving an AAAA record (for this server) and no IPv4 records (+/- DNS timeouts, of course...).

Tomorrow I will work on getting a nameserver or two running on IPv6, to extend the IPv6 infrastructure another level.

This is all me fiddling now that we have an IPv6 allocation at Catalyst, and are developing our IPv6 infrastructure in order to be able to deploy a couple of the .nz nameservers on IPv6 in due course. I'm not sure if I can point a .org domain at IPv6 nameservers so I might have to pick one of my .nz domains and use that to have a top to bottom IPv6 infrastructure.

For IPv6 to start to work, I think people just have to start using it, so I'm going to change the systems that I can change. So I've also put a fixed IPv6 address onto my laptop, which is nice, and I will be converting all of my home systems to IPv6 addresses as well as time permits.

Of course I can't get rid of IPv4 yet, but I think I will probably be trying to run that through a proxy somewhere non-local if I possibly can.

If you can access this server via IPv6 I would be really interested to see a traceroute from you :-)