Linux – Hot clone a living Linux service

clonelinux

We need to hot clone a Linux service when it's alive, not just because of we can't reboot or something; it's just because of our special scenario (yeah, I've already read this answer, but it's a little bit different from mine Clone a working Linux server).

We have a calculation node, you can say an NLP calculating node which is running some models on it. When we start the node (with a service of course), the calculation will be horrible slow until we feed it several times. We called it warm-up.

Unfortunately, the warming-up job takes a long time for us to wait for (maybe our calculating finished before the node warmed up).

So, the problem comes, is there a stable way to hot clone out a Linux server to keep the node at the best performance so that we can clone and make it online in a shorter time?

Best Answer

Maybe you can't "hot clone" a whole server (you can, but only if it's a virtual machine), but you can freeze and restore a single process, with criu, Checkpoint/Restore in Userspace.

This allows you to save the program's internal state to disk and stop the program, and later, to restore the program to that state from the saved files.

To support your desired operation, you can copy the files representing the saved program to another server, and restore it there.

criu requires a recent kernel with various features compiled in, so older Linux distributions might not work. You can run criu check on a particular machine to determine if the prerequisites for criu are present.

Related Solutions

Linux – Clone a working Linux server

There are many ways to copy disks, file systems or files. Generally, copying the file system gives you a good clone with the flexibility that it can be a slightly different size on the target system. With the target system running some sort of live Linux (knoppix, ubuntu live, etc.), booted from a cdrom, you can create the partitions on the disk using fdisk or your favorite partition application. Assuming you have an SSH server running on the source system, take an approach similar to this:

http://www.linuxfocus.org/English/March2005/article370.shtml

The magic is happening in this command:

ssh sourcePC 'dump -0 -f - /' | restore -r -f -

When using any particular method to clone disks on a live Linux environment, your only concern would likely be with the databases. The best way to backup and restore a database is to use their dump tool to make an ascii file snapshot of the database just prior to the file system dump. For mysql there is :

mysqldump --all-databases > mysql_databases.sql

For postgresql, there is:

pg_dumpall > pg_databases.sql

If you encounter any sort of consistancy error on the new system, restore the database. Alternately, once you have shut off services on the source system, do the DB dump again, and restore on the target, and you will not miss any recently modified data.

Linux disk clone: tar vs special clone utility

The third option is to dump cloning and instead use a proper system configuration management tool such as Puppet or Chef. Cloning is a really bad idea for systems that you need to maintain over time, as you need to apply changes to all machines currently in the field, as well as respinning all of your clone masters. If you use a proper management tool, though, you just describe the state you wish a system to be in, and then the tool makes sure that the system is in that state -- whether it just came "factory fresh", or has been in production for several years and just needs to have a config file tweaked.

Basically, your new machine process should be:

Use the OS' native automated installation procedure (d-i preseeding works really well) to get a base minimum system installed that is capable of running your automation tool (and nothing else);
Run the automation tool to configure the system to your liking.

Best Answer

Related Solutions

Linux – Clone a working Linux server

Linux disk clone: tar vs special clone utility

Related Topic