Linux – Atomic copying of multiple directories

deploymentlinux

This question is regarding deployment of web applications.

Intro (you can skip)

I'm using django, and the way my hosting provider have set up their django support ends up causing the webapp to be scattered in at least three locations:

  • Actual application code in /something/<my_apps>/
  • Site settings/urls/templates in /something/<my_site>/
  • css, javascript, and other "media" in /something_else/media/

So when I deploy/upgrade the website, I need to update several directories at once.

Actual question:

Is there a way to make an atomic file copying? I'm not "expert" at linux systems by any means, so please forgive my ignorance.

The copy operation involved several directory trees, either two or three, basically:

copy _tree1 to tree1
copy _tree2 to tree2

By atomic, I mean:

  • It's either fully copied, or not copied at all. It should never be in a state of some-copied-but-some-failed.
  • It's done in the shortest amount of time possible. Ideally there should be no point in time where the system sees the copy as it's happening in progress, it either sees the old version of the files or the new version; at no point in time shall it see the old version of file A but the new version of file B. If that's not entirely possible, then it should take no more than a few milli seconds.

My idea is to have something like double buffering: I prepare everything in a staging area, for instance, _tree_x, and then copy move it to tree_x with should be an atomic operation that simply changes pointers on the disk.

I think that a single such copy move operation is atomic in linux, (is it not?), but I need several such operations to be atomic as well; I want them to be treated as if they were a single move operation.

Best Answer

I think you are on the right track with a staging area. I'm not aware of any atomic commands, but if you stage your files and then use a script to remove the first directory and move (not copy) the second, and do it for all three directories, it should be really quick.

Alternatively, you may want to use symlinks. That way you could have roughly:

/version/22/<my-apps>
/version/22/<my-site>
/version/22/<my-media>

and deploy a /version/23 directory with the same subdirectories. Then, where the actual file would go (and again, for speed, you'll need a script), you can use a symbolic link so that when anyone goes to the latest page, they get whatever version is current (and it all happens transparently and they have no idea). The benefit of this is that your older work is still around until you decide to delete it. [Although, granted, a versioning control system is a better thing to use for keeping older work.]

You'd have to check that 1) you can run scripts (and in such a way that web users can not!), and 2) that you can use symlinks (as some web servers are configured to not follow them.)

Related Topic