Using incron to autocommit changes in a folder

A friend e-mailed me this morning asking for some help with a problem he had where he wanted to make a folder writable by a group of people without making the files deletable. Stepping back from his question, I first pointed out that if the files are editable then they can be effectively deleted by removing the content from them, regardless of whether the directory entries themselves are retained.

One solution which occurred to me would be to automatically version the content of the directory, and this reminds me of why versioning of /etc has never worked for me: it only happens when I remember to commit.

Normally when I edit files in /etc, I am focused on achieving something now, and not on being able to undo it later. To this end I have written myself a script which will commit the contents of a directory (including adding new files and removing deleted ones) into a git repository, as follows:

#!/bin/sh
#
# Allow for debugging.
[ -n "$DEBUG" ] && set -o xtrace

cd $1

MESSAGE="Autocommit"

# Initialise the directory if it's not in Git currently
if [ ! -d .git ]; then
  git init
  MESSAGE="Initial commit"
fi

# Any files that aren't in the directory, but are in the repository can be removed.
for F in `git ls-tree HEAD | cut -f2`; do
  [ -e "${F}" -o -L "${F}" ] || git rm "${F}"
done

# Any files that are in the directory, but aren't in the repository can be added
git add .

# And commit...
git commit -m "$MESSAGE"

So this script takes one parameter: the name of the directory where changes have (presumably) occurred, and just blindly commits everything there.

The magic glue, then, is the awesomeness that is 'incron' :-)

My first target is the ~/bin directory of all of those useful little scripts that I tend to randomly edit without nearly enough version control:

incrontab -e

home/andrew/bin IN_MODIFY,IN_DELETE,IN_CLOSE_WRITE,IN_MOVE \
     /home/andrew/bin/commit-directory /home/andrew/bin 2>&1 | logger -t andrew-bin

With this, it means that whenever a file changes, is deleted from, moved into, or out of the directory we are monitoring, our commit-directory script is run.

Just for testing I delete some ancient files that should have been killed a few computers back. A quick 'git status' in the directory shows everything was committed nicely. I edit a file and make some quick changes. Once again, I go into gitk and can see all the history.

I sit back and ratchet my laziness another notch with a nice cup of tea, happy that, for now, at least, changes to my scripts are under revision control at last.

Outstanding Issues

Well, ok: it's usable as is, but there are a few small things that could be improved:

  • When I edit a file, the temporary swap file gets committed, and then deleted as the changes get committed
  • If I make a whole lot of changes the script can fire multiple times, or even fire before some changes fully happened
  • Maybe I want a shadow folder maintained and committed, rather than just this folder
  • Perhaps it would be a good idea to push the repository elsewhere

I'll keep monitoring in the real world if these are real annoyances, and whatever other annoyances I find, and maybe I'll tweak that commit script a bit more - add a delay and/or some locking too it. I don't think that adding some push would be too big a deal either, and I'd love to get suggestions from Git geeks as to how to do this more simply/reliably too.

Temporary files

'git add .' will not add ignored files, so defining .gitignore or .git/info/exclude to exclude the editor temporary files should trivially deal with that.

The partial changes problem would be best resolved in the incron. When changes happen, it should have an optional timeout and only run the command after that, so if more changes happen in that time, they would be handled cumulatively.

gitignore & accumulation

I added a .gitignore pretty quickly in fact, but didn't get the chance to test before I let my blog out into the wild. I also wasn't sure how well it deals with ignored files that are already in the repo, or with removed files.

I had thought about doing the cumulative changes thing by doing something like:

grab a lock or exit

until no changes in last 1 second do:
 wait 1 second
done
release the lock

commit the directory

I think that would work, but in fact it seems kind of overcomplicated for what I really wanted to achieve. What I have done is easy to understand, and gets it right in 'most cases', so I stuck with it 'as-is' :-)

Cheers,
Andrew.

give it a longer delay :-)

The grab-a-lock trick will work well (dotlockfile has just the right options :-)) . Just give it a longer wait time.

Note for casual readers: this strategy won't work with subdirectories. Incrond and all inotify (idiotify?) tools are per-directory without recursion.

On the XS, I use incrond a lot these days for the kind of privilege escalation where you'd have a priviledged daemon listening on a socket. The 'priv daemon' model is harder to program and eats RAM (for many such single-purpose daemons in interpreted languages).

So what I now do is define a directory in /var/spool/foo/request (perhaps protected by ACLs or POSIX access control) where you write your "message" to trigger the priviledged script. Responses (if any) are stored a few cycles later in /var/spool/foo/response.

I used a lockfile approach

I used a lockfile approach for another incron-based script to replicate a CVS repository.

As for not launching the script twice, what you need is IN_NO_LOOP, e.g.:

/var/lib/gforge/chroot/cvsroot/mantis/CVSROOT/history IN_CLOSE_WRITE,IN_NO_LOOP nice -10 /usr/bin/cvs-syncer $@

will not call cvs-syncer again before it returns.

Very nice, I was looking for

Very nice, I was looking for the equivalent of subversion's autoversioning feature[1] for some time now, and this comes pretty close.

[1] http://svnbook.red-bean.com/en/1.5/svn.webdav.autoversioning.html

Autoversioning

Given what incron is actually doing, it is not too hard to envision writing a (e.g.) perl script to run as a daemon monitoring all changes in the directory and commit each one to Git as an individual unit.

I picked the above example more as an idea of a cool hack using incron, which I often find to be unknown or poorly understood. It's supposed to make people think about other system administration tasks that it could apply to...

Cheers,
Andrew.

look at etckeeper

etckeeper is another of Joey Hess' little gems.

You should glance at it for two reasons:

1) It addresses the problem of tracking permissions -- it creates a script called .etckeeper that contains the mkdirs, chmods and chowns required to restore the /etc directory tree including permissions and empty directories, and checks that script into git (or whatever)

2) It addresses you "I always forget" issue to some extent by automatically committing by hooking into apt and doing a commit just before and after every package installation/removal.

Cheers, Phil.

Thanks

Your fine blog entry just helped me to get my emacs org-mode files versioned.

Greetings from Dresden, Germany

Recursion?

Having played around some more, I have realized a MAJOR bug with incron .. its not recursive. This means that if you change /etc/apache/somethingelse it wont register it. It will only register things in /etc/ alone etc.

Only sees specific directories / files

Yeah, I guess that is a limitation. Not sure that I'd see it as a bug though. Essentially it monitors files for change, and one type of file it can monitor is a directory, so realistically I think that is 'working as designed' :-)

I believe it's also quite possible to write something to hook into the lower level libraries that might expand it's monitoring as it discovered new directories, and so forth.

Maybe submit a patch to incron with a new option that told it to monitor recursively. I certainly wouldn't want it to do that by default though.

Cheers,
Andrew.

Also look at flashbake

Flashbake, available at http://bitbucketlabs.net/flashbake/, seems to implement the wait-and-timeout thing that you mention. I also suspect it monitors subdirectories.

Ethan

[D] [Digg] [FB] [R] [SU] [Tweet]