I've been working on implementing CardDAV support into DAViCal at the moment, and the first problem I encountered when I went to try and use it from iCal, was that the configuration on iCal didn't seem to want to let me enter a URL to my addressbook.

For several years now we've been buying our groceries online. It's worked well, and for the last couple of christmases I remember Heather adding a six-pack into the pre-christmas order so she could pull it out and hand it off to the delivery guy.

Fair enough too, because he was their front-line man. He was the guy who had to actually meet the customer, and even if only for two minutes face time, the impression he gave with his cheery "seeya mate" on the way out, and his always-happy smile, was that getting the groceries delivered was fun.

A friend e-mailed me this morning asking for some help with a problem he had where he wanted to make a folder writable by a group of people without making the files deletable. Stepping back from his question, I first pointed out that if the files are editable then they can be effectively deleted by removing the content from them, regardless of whether the directory entries themselves are retained.

One solution which occurred to me would be to automatically version the content of the directory, and this reminds me of why versioning of /etc has never worked for me: it only happens when I remember to commit.

I released a new 0.9.7.2 version of DAViCal yesterday. This reflects quite a lot of stability and small fixes for some subtle problems, and quite a lot of work with the iPhone, adding the possibility of a simpler configuration experience for iPhone users.

For several years I've wanted to join the Calendaring and Scheduling Consortium and go to one of their events to get a chance to meet face-to-face with some of the luminaries in the calendaring world, but every time there is an event it seems to conflict with either linux.conf.au or my brother's wedding or something. Finally I've decided I can make the next meeting, so I've paid over the money to join the organisation and I'm travelling to the US next month for 'CalConnect XVI'. With that on my mind when I saw an HP 110 mini netbook on sale for NZD$588 from Harvey Normans I finally flipped over the 'shall I get one' threshold, hoping it will make a good 'travel laptop' for the upcoming trip.

Something that has been annoying me recently with my bank has been that their website tells me that they will never ask for my password over the phone. And then their call centre asks me for my password. Over the phone. Of course the call centre doesn't mean my website password - they mean the special 'ultra-secure 5ekr1t code phrase', but they don't have a good, universally understood word to use for that. Hopefully they'll work one out, but they appear to have got the message anyway.

This got me to thinking about how these phrases are used, and how insecure they are in reality. After all when I store a website password I go to significant lengths to ensure that the same password is not represented by the same string of characters in my database. How vulnerable are our secrets in the databases of organisations we do business with?

Yesterday I switched my development environment to PostgreSQL 8.4, and so today I foolishly used the PostgreSQL 8.4 manual while I was developing, without thinking that I might be using some new functionality. Silly me!

What I wanted to do was to convert a column of words into a comma-delimited list (for readability, not for export), to get output something like this:

 id  |                   tags
-----+----------------------------------------------
 141 | DAViCal, FOSS, Programming, CalDAV, Releases
 138 | Family, Life, Kids
 137 | Kids, Family, Rants
 136 | Life, FOSS, Debian, lca
 135 | Releases, FOSS, Packages, Debian, DAViCal

Where the table has two columns 'id' and 'tag', like:

 id  |     tag     
-----+-------------
 141 | Releases
 141 | Programming
 141 | CalDAV
 141 | FOSS
 141 | DAViCal
 138 | Kids
 138 | Life
 138 | Family
 137 | Kids
 137 | Family
 137 | Rants
 136 | Debian
 136 | lca
 136 | Life
 136 | FOSS
 135 | Packages
 135 | Releases
 135 | DAViCal
 135 | Debian
 135 | FOSS

I looked at this and thought: that's just the job for an aggregate function! It's like sum(), except it concatenates!

I was going to reply to this in the mailing list, but after I spent half an hour furiously typing this, I thought it might be better to put it into a blog, for a slightly wider audience, perhaps a more permanent life and partly in an effort to end this thread on a more or less constructive note.

The backchat for this is that the LCA 2009 video has started to go up on the web, and some talks have some gaps in the audio stream... Russell Coker wants to make a constructive suggestion, as follows:

On Wed, 2009-02-25 at 18:02 +1100, Russell Coker wrote:
>
> I've pasted in the below paragraph (the third time I've included it in this
> thread) to reiterate my point. Note that I am not asking you to re-do any
> videos, I am making a suggestion for future people.
>
> # I suggest that in future when preparing such videos the sections with no
> # sound be omitted and replaced by a short note explaining the omission,
> # something like "sorry, due to technical difficulties four minutes of sound
> # were not available". Having a second ogg file for the lip-readers would be
> # an option although I expect that the demand would be minute.
>

As someone who has not been at all involved with the production of the LCA videos, but who has been intimately involved with the process of streaming DebConf 6, 7 & 8, I think this sort of additional effort will only happen if people voluntarily do it themselves.

Just the task of *watching* all of the produced videos is a huge job for one individual, but as a community of individuals it is quite likely that at least one individual in the community watches each video. If each person who was offended by the quality of post-production work took it upon themselves to fix up only one video then it is probable that we would see some changed. Otherwise I'm afraid it is unlikely.

Further in his e-mail, Russell Coker wrote:
>
> My observation was that the microphones were put down AFTER they ceased
> operating. A non-functional lapel microphone is no better than a non-
> functional hand-held microphone.
>
> Having a large supply of new batteries would be one way of alleviating the
> problem (I believe that some of the instances were due to flat batteries).
> Another possibility is having two microphones on hand so that if one died the
> other could be turned on.
>
> A wired microphone that doesn't rely on battery power would probably be the
> most reliable option. That of course might not fit with OH&S issues.

Most speakers are uncomfortable / unfamiliar with wired microphones. Spares are essential, of course, but the problem can usually be avoided by having a checklist for the audio person to confirm several things:

Before the start of the talk:

• check that batteries are good enough to last for the full length of the coming talk.
• check sound levels are correct for this speaker
• confirm sound is going through to recording
• check the speaker knows how to turn the mic off/on

At the start of the talk:

• confirm sound is going through to recording

At the end of the talk:

• Turn the mic off to save battery.

There are probably a couple of things that I've missed, but if you start with a checklist you pretty soon modify it into a *good* checklist, and it *really* helps when people are under fire.

In fact lapel mics are not the best microphones for these kinds of presentations. The best mics are the ultra lightweight 'headset' models which place a the mic near to the speakers mouth. These do not suffer when the speaker turns their head hard to the left or right, at which points a lapel mic stops getting their speech. They also work well with that more hirsute minority so over-represented in our particular community (alas, that this set no longer includes Bdale :-)

Of course DebConf has a large team of volunteers for streaming the conference, and has developed these kinds of procedures over a number of years. At DebConf there are usually only two main streams, each of which involves:

• The director, operating the video mixer
• Speaker camera operator
• Audience camera operator
• Slides to video convertor
• Sound mixer operator
• Talk timer, to warn speakers at t-10, t-5 & time is up.
• Two roving people responsible for getting audience mics to people talking in the room.

Ideally that really is 7 people (times two streams), and you can get by if you can't find all of the last three, but the other four are increasingly desirable. We mostly manage to do that with volunteers for two streams at DebConf, but for five or more streams at LCA it is inevitably a lot harder, and the quality necessarily has to be cut back to match the resources available.

While it is true that sound is critical to this communication, it is unfortunately also true that most people's (even most geek's) eyes glaze over when presented with a mysterious box with a mere 6 sliding potentiometers, let alone when there are 36 of them, each associated with another 8 rotary knobs, and a rats nest of cables worthy of the worst network nightmare they've seen, and where everything appears to be literally held in place with duct tape.

So it is not so surprising then that while it is relatively easy to find a gadget mad geek capable of operating a camera, or even experienced with operating a camera, finding people with experience operating a sound mixer is an order of magnitude harder.

At LCA the operation appears superficially to often involve a (single) speaker camera operator with a very basic sound mixer which has been configured once by an overworked person who is unobtainable during the actual talk to solve any audio problems. These people are volunteers, and are doing their best, but it simply isn't possible to get a consistently high quality of video and audio in those circumstances.

So as one of the organising committee for LCA 2010 in Wellington I will be watching what we do, and although I don't intend to get personally involved in the video production (I've got plenty of other stuff on *my* plate :-) I do intend to provide what advice and assistance I can. I know that we hope to have some professionals involved (as volunteers), but how many volunteers, and with what levels of skill & experience, we are unlikely to know until much closer than the date. I'll also personally try and get a run-down on all of the audio equipment so that in an emergency, if I happen to be available, I can sub as a sound mixer operator.

So we will try and do better in Wellington in 2010. Come to the conference, though, because we can make no guarantees, and if you do come to Wellington, and you see a single cameraman, and you know something about running a sound mixer, then perhaps you can come and volunteer to help out in that capacity, for the talks you are attending anyway. We'd really appreciate the help.

And finally I must say thanks, in particular, to Holger Levsen for all the learning I have gained since joining the videoteam at DebConf5 in Helsinki in 2005. And too, to all of those past conference teams (LCA, DebConf and otherwise) who have put their best efforts into providing videos of the talks I couldn't see in person. I know it's a bloody hard job, and often a less high profile one, so:

Thank You.

When Heather designed a logo for me for Morphoss she did it with a bitmap editor, naturally enough because that's the tool she's most familiar with using. I'd rather not use a bitmap as the source format for the logo though, because it will degrade when it gets resized, so I redrew it as a vector graphic.

One of the best free, open-source tools around for vector graphics seems to be Inkscape and I've mucked around with it for many years, so I naturally used that.

Once you have a logo though, you naturally want to use it in documents, and the importing of SVG graphics into OpenOffice.org documents is a long-outstanding bug (let alone embedding SVG graphics) so I needed to convert them to another format. It's actually the most requested feature in OOo, appearing twice in the top 10, and even spawning an external SVG importer project.

Since both programs support encapsulated postscript I was able to save the logo from Inkscape as .eps and use it directly in OpenOffice.org. While this initially seemed satisfactory, after a few weeks of using documents with the .eps logo embedded in them I started to get annoyed with the strange pauses when my CPU was maxed out while paging up and down. I was sure that that had not happened in the past when I was using a logo in WMF format, which OOo inevitably has to support well for compatibility with other Office Suites.

After some searching around for more complicated ways to convert SVG or EPS to WMF, I discovered that what I could do was simply to open the EPS in OpenOffice.org draw, and save it from there as a WMF. This seems to work well, for my purposes anyway, so now when I use my logo in my OpenOffice.org documents I don't see any annoying slowdown paging up and down within the document, and I didn't have to download the SVG importer for OpenOffice.org either.

Well alright, I did download the SVG importer as well, but my logo didn't look nearly so good without it's text, and with everything displaced up and to the right at various offsets!

After a few years of only buying laptops with Intel hardware, today I bought something totally different. It's not really what I wanted (which was an HP HDX 16t) but I get the feeling that none of these 16" HD 1080 laptops will make it to New Zealand for a while yet, and the NZ dollar has done such a nosedive recently that it's better not to wait any longer.

In the places that hold stock there seem to be some good specials around at the moment, and as the owner of a new free, open-source consulting business (i.e: a cheap bastard) I went shopping for the cheapest dual-core I could find with a half-decent screen, and I found the Asus X53K for $999 (USD$589) at Dick Smith, including a 2G ram upgrade to take it to 3G. It's entirely non-intel, with a 2GHz Turion dual-core, ATI Radeon X2300 with 1440x900 panel, Atheros AR2425 wifi and 160G HD. I'd bought a replacement 320G hard drive even before I got the laptop, so now I have a pristine, unbooted 160G hard drive with the install files for some other OS on it - no doubt I'll find a use for the disk, at least!

Since AMD got ATI to release all their chip documentation earlier this year I felt able to shell out for this, rather than the extra $100 for the model next to it, and it was nice too to get home and find that Atheros have recently released the HAL for their a/b/g chips. Which presumably means that they haven't done so for their 'n' chipsets, and I should continue to steer clear of that technology for a while yet...

I'm running Debian GNU/Linux 'Sid' on the Asus X53K and, everything pretty much just works out of the box. My installation process was to rsync the old laptop onto a new disk, and boot the new laptop from that - after compiling a new kernel more appropriate to the changed hardware.

After overcoming my own stupidity in not syncing the /dev/ underneath udev, which I easily googled my way out of, the only problem I've found so far is that the free radeon driver doesn't do 3d for me. Presumably the non-free ones would, but they won't compile against my 2.6.27 kernel so I don't know for sure. Fortunately I don't use 3d for anything so it's not a huge inconvenience to me. With 3G RAM and a fast 320G hard drive the laptop actually is an upgrade for me, too, and it has a webcam too, which I expect I'll look at in much the same way as I did the fingerprint reader on the old laptop. It will be good to finally hand that old one back to Catalyst, too, who have given me the flexibility to take my time on this.

Now to try and peel off all these stickers without damaging anything!

Writing free, open-source software is an incredibly public activity. Everything you do is in the public eye, and google will inevitably discover your site, and then other people will find your software, and download it, and this is a good thing. It's why you're doing it, after all, and it's so nice to receive those occasional 'Thank you for your software' e-mails. There are occasional exceptions, however.

Today's practical exercise is to demonstrate your skills responding to the annual student exercise question, like this one, following on to finish a real exchange while still retaining your sanity to the maximum extent possible. Humour will receive bonus points.

Here goes. First up, we have an e-mail arriving out of the blue which looks like this:

how to run the caldav server
in window
i have download it from the http://wiki.davical.org/

The IPv6 wave progresses apace. Well, perhaps not 'apace', but it is moving...

The latest kernel exploit has incidentally had some local fallout in causing more of our boxen here to be upgraded to kernels with IPv6 support, and as a consequence our mail server is now reachable on IPv6. Some have suggested that making it only reachable on IPv6 is a good solution to spam but I suspect that there are still a few mailservers out there that we do want to receive e-mail from which are not IPv6 capable yet!

Since I can now SMTP and IMAP happily over IPv6 I decided it was time to get more adventurous. IPv6 is now in Squid3 head, so I built Etch packages of that and it seems to be 'basically working' in a few places now. We've been using ircd-ircu for a long time for an IRC daemon and it similarly seems that now has IPv6 support, so I backported that to Etch as well.

Packages are available for i386 and amd64 from my repository:

deb http://debian.mcmillan.net.nz/debian etch ipv6
deb-src http://debian.mcmillan.net.nz/debian etch ipv6

If I think of more Etch things that I need for IPv6 I'll put them there too. I do have dircproxy for Etch with support for connecting to IPv6 ircd but I seem to have misplaced the packages somewhere. If you're keen on seeing that then I'm sure I can reconstruct them somehow...

Now that we are having increasing amounts of IPv6 around some things are starting to reduce down to a 'Small Matter of Firewalling', which is suggesting to me that we will need manage our firewall rulesets differently for IPv6 than we have for IPv4.

In a lot of cases we can turn on/off large chunks of access related to a particular person/organisation by disabling a VPN, with the firewalling being a somewhat static monolithic overriding control above that. With the control potentially moving away from the VPN, and more directly into the firewall rules, we will need clearer association mechanisms in place. Of course we will continue to have VPNs, but they might become somewhat simpler, reducing in many cases to encrypted tunnels between exact endpoints.

A GPS is one of those toys that I have wanted for a very long time. So last year I finally marshalled enough excuses in one place, lined them up and plunked down some money for a Garmin GPSmap 60CSx in the vague belief that there's enough Linux software out there which understands GPS, and the Garmin is a brand that seems to have mature Linux support. And besides, I'd been told it was a good model that would do what I wanted and then some.

Once I got it I naturally wanted to do stuff with it, and in particular I wanted to connect it up to my laptop which (of course) is running Linux and I found that it wasn't nearly as trivial as I had thought.

This prompted me to run around finding software to use with it, so here's my capsule review of my journey from "extremely naive" to "very naive". Perhaps I'll also learn something from the comments of people who are further down the track.

My first resort was "apt-cache search gps", of course, which immediately brings up such fine sounding programs as "gpsman", described as "A GPS manager" (I haven't managed to get it to do anything yet), "gpstrans" which since it specifically mentions Garmin GPS sounded like just the ticket (it appears to be quite old and superseded). There was some useful stuff as well.

Getting Maps on the GPS

Since the Garmin GPSMap series will display maps, I wanted to be able to get some mapping data on there. There are two map sources I was interested in:

• Map data from NZ Open GPS Maps which gives comprehensive coverage of New Zealand, and is free and community maintained, albeit with some annoying uncertainties about the licensing of the data, and it's being tied to Garmin GPSs.
• Map data from OpenStreetMap.org which gives excellent coverage of many areas of the world, and is free for use under a creative commons license.

Using the NZ Open GPS Maps

The NZ Open GPS Maps are built specifically for the Garmin GPS, so getting them onto my GPS was relatively simple once I found out the exact command line was remarkably hard.

The program that I needed to find to be able to get the maps onto my GPS was sendmap20 which is a free (as in beer) download from the cGPSMapper Website.

I downloaded the maps from the NZ Open GPS Maps hosted on the cGPSMapper site. I needed the files identified as 'binary' files - the installer is a Windows program and no use on Linux. They all have filenames like 64000012.img.

The command-line I first used to install the maps onto the GPS was:

sendmap20 -t/dev/ttyUSB0 640000??.img


That works fine, but if you have even a moderately sized micro SD card in the GPS there are much faster ways. As I found more free maps I found much better ways to do it.

An alternative approach

In the GPS (well, in my one anyway) there is a micro-SD card which is FAT formatted and all of the installed maps are in one humungous combined image of all of the uploaded maps. This means that there is no way to straightforwardly add/remove maps without creating that image, and re-uploading the whole thing. The image file is called 'gmapsupp.img' and is in the 'garmin' subdirectory on the SD card.

This means that you can create the file (or even several different files, with different maps on them) and move them directly on there much quicker, either by taking the SD card out and using it in a USB2 reader (for really big files) or switching the GPS to operate in USB Storage mode (which is USB1, but OK for smaller files).

You can create the IMG file to copy in as gmapsupp.img with sendmap20 also, with a command-line like:

sendmap20 -lNZOpenGPSMaps.img 640000??.img

and then, if you mount the GPS as /media/GPS, you can copy that file on directly

cp NZOpenGPSMaps.img /media/GPS/garmin/gmapsupp.img


QLandKarte

Another program I found useful specifically for dealing with the Garmin GPS Map format is QLandKarte, which understands the native Garmin format and will display the maps in a graphical window.

You can also point and click to select maps, building up a specific set that you can then upload to the device from within QLandKarte itself. This seems to operate a lot faster than sendmap20, presumably because it's driving the USB directly, rather than through usb_serial. To get it to work I seem to have to (a) sudo rmmod garmin_gps and (b) sudo QLandKarte, however, which is definitely not ideal.

For the moment I will continue to use sendmap20 to create files which are sets of the maps I want, and will put them on the device in USB storage mode.

QLandKarte is still very useful, however, for looking at maps and deciding if they are worth bothering with. I used it in this way to select the Australian maps I took with me when I went to Linux.Conf.AU recently. There are more free Garmin GPS maps.

OpenStreetMap.org

Those Australian maps came from OpenStreetMap.org and ultimately I expect that it will become the best source of data for creating maps for my GPS. OpenStreetMap.org is an attempt to provide community-maintained maps for the whole world and which seems to have made significant progress in UK, Europe, America and Australia as well as many other parts of the world.

The data from OpenStreetMap.org does not have the same licensing restrictions as are present even in the NZ Open GPS Maps data (which is relatively free, but has problems similar to the old-style BSD license).

There are also some good mapping interfaces to the OpenStreetMap data although the searching currently still leaves something to be desired. Interfaces are also available for extracting subsets of the data, which is in a fairly straightforward XML format, or you can download the whole lot, but at nearly 3GB for the bzip2 compressed version you won't do it every day.

I gave a brief overview talk about OpenStreetMap.org while I was at Linux.Conf.AU and it seemed to go down quite well (I gave it several times, in fact). You don't need a GPS to contribute to the project, and of course the maps which are being created are usable for many purposes beyond their usefulness on GPS. Some people apparently even print them out, but that just seems weird!

I hate spam!

Which probably puts me in the same camp as 99.99999% of the world. The other 1 in 10 million are, of course, the spammers, who seem to take the space invaders approach to sending e-mail: we'll keep sending you more until you die.

A few years ago I used to only receive perhaps 1 every 100 seconds, which was pretty annoying, but Spamassassin was quite able to filter out 99% of those and let through about 1-2 each day, which I could deal with. My spam levels increased to maybe 1 every 20 seconds, and late in 2005 I implemented a second layer of spam filtering on my laptop using DSpam. This worked quite effectively, but DSpam is really not the tool for the job - it's much more appropriate as a company-wide antispam solution, and potentially as a replacement for Spamassassin. It drove me nuts on my laptop because it's resource usage slowed down the interactive response.

When I got my new laptop at the beginning of the year I decided against continuing with my rather baroque mail setup and to leave the spam filtering on the server. What I didn't realise is that my spam rate had increased again to around 1 every 8 seconds, and it has been slowly driving me to distraction ever since. It seems to have cranked up another notch recently, to perhaps 1 every 3 seconds now, so that 1% making its way through Spamassassin was getting to a very annoying several hundred each day. The longer I took to resolve it, the more time I would be wasting dealing with it every day.

What I chose to apply on this occasion was CRM114, which I had some vague idea might be able to help. I was fairly impressed by the relatively simple install, but what completely blew me away was the speed with which it was able to learn to be useful. Starting from scratch, it seems to be correctly classifying over 90% of my incoming mail after about 12 hours of training, on a total of only 75 'Unsure' messages. Even after only an hour it was getting over 50% (I'll describe my actual CRM114 installation process in a comment below). So far there have been no false positives.

Now that CRM114 is installed I will be able to look into some of it's other mail classifying features too, and I'm really looking forward to that too.

The availability of IPv6 worldwide is surprisingly extensive, nowadays, but over the past years as it has slowly filtered around, people have had bad experiences with it because of poor routing. It seems that, as always, a bad rep travels about 20 times further than a good one, so an automatic response to casual problems that people see when using IPv6 is to blacklist it, without actually investigating the problem.

Take today, for an example. Someone said to me "I disabled IPv6 in Firefox because it was slow for one of my favourite sites". OK, so show me this favourite site. Show me the traceroute. Give me some facts!

Further investigation showed that although there is an AAAA record for www.crooksandliars.com, there is nothing listening on the other end! Looking at the AAAA record returned, we see that it is an autoconfigured IPv6 address of the form xxxx:xxxx:xxxx:xxxx:xxxx:xxFF:FExx:xxxx and can conjecture that the likely problem is that someone has done a hardware upgrade on their server, so they now have a different MAC, and consequently the autoconfigured IPv6 address has changed. Other scenarios are entirely possible, of course, but this is a likely one.

This is the second case I have seen where someone was running publicly available services on IPv6 using a manual DNS record pointing at an automatic IPv6 address, but I doubt that it will be the last. Unless there is infrastructure in place to automatically update your DNS when your address is autoconfigured, you are going to get bitten by this problem at some point if people remotely connect to your system for some service.