Bacula: initial Full Backup job taking too much time

backupbaculaslow-connection

I've chosen Bacula (v.5.0.2) to do backups of our office worker's data.

The Director and Storage Daemon run on a Debian based root server (the SERVER) rented from a major german hosting company.

All the precious data and therefore Bacula's File Daemon (the CLIENT) resides on a Debian based Samba server that's located on site and connected to the internet via ADSL (~768Kbit up) behind a SOHO router (it's a non-profit organization that's cronically short on money).

Moreover, the CLIENT's internet connection is being disconnected by the ISP every 24 hours and its IP does change every now and then.

I've managed to set up the basic infrastructure and successfully tested backing up and restoring files. But, now that it is time to define the automated jobs, I am facing a problem that I just can't get my head around:

Problem

Assume that I want to have a job per office worker and need to backup his Samba share which weighs around 10GB. The backup level does not matter as Bacula needs to do a Full Backup to start from.

But this initial Full Backup won't make it across the office's wire before a forced disconnect renders the job useless.
Question

Is there any way to partition the initial workload into smaller pices other then setting up multiple jobs (and file sets)?

Or better, is it possible to somehow pause and restart a running job to overcome the inevitable connection loss?

Anything else I didn't think of that might help?

Best Answer

Backups take an amount of time proportional to the data being backed up -- If you need to move 10G of data it's going to take some time (how much time being determined by the speed of your network link).

Further bacula assumes that your network connectivity is reliable -- it doesn't expect your connection to drop, and frankly that's a reasonable assumption on bacula's part.

The PROPER solution to your problem is to change ISPs.
Frankly if your network connection is as unreliable as you describe (disconnected every 24 hours!) and you're not getting the connection 100% free of charge you're probably overpaying.

If you're not able to implement the proper solution, your best alternative is to run a local Storage Daemon at the client site and back up to disk there.
You can then use a tool like rsync to copy the virtual tape files to another location.

The major disadvantages to this solution are that you'll need (at least) an extra disk to back up to, and you'll have to manage the rsync job (which you can do with a RunAfterBackup script in Bacula).
Depending on your needs a USB hard drive may suffice for the extra disk.

The other alternatives you described in your question are either flawed or not possible.
Specifically, Bacula has no concept of "pausing" a backup, and splitting the backup into multiple jobs (the only way to break it up) just means you'll fail a sub-job rather than the entire thing -- you might still wind up with a backup that doesn't restore you to a good/consistent state if you do this, so you're not actually addressing the problem.

Related Solutions

Bacula virtual backup job doesn’t run, no output

Type "messages" in bconsole and it will output all logs from jobs, but just once. Another way is to configure bacula director with good email, and it will send logs to that email.

Bacula off-site backups

I found (a) the solution...
it seems that the key is here:

It occurs automatically, if the source SD and destination SD of a Copy or Migration job are different.

My goal was to copy some backup jobs from Asia to Europe. In order to do this I have created a Device in the Europe SD:

Device {
  Name = asiaBackup
  Media Type = File1
  Device Type = File
  Archive Device = /bacula/asia
  LabelMedia = yes
  Random Access = Yes;
  AutomaticMount = yes;
  RemovableMedia = no;
  AlwaysOpen = no;
  Maximum Concurrent Jobs = 40;
  Autochanger = no;
}

Then in the asia director I have created one storage that points to this device from the europe SD.

Storage {
  Name = europeStorage
  Address = myEuropeDomain.com
  SDPort = 9103
  Password = "blaBlaBla...."
  Device = asiaBackup
  Media Type = File1
  Maximum Concurrent Jobs = 5
  Heartbeat Interval = 10
}

/* dbPool = the pool where the backup sits locally */
Pool {
  Name = dbPool
  Label Format = "dbBackup"
  Pool Type = Backup
  Recycle = yes                       
  AutoPrune = yes                     
  Volume Retention = 365 days         
  Maximum Volume Bytes = 5G          
  Maximum Volumes = 1000             
  Label Format = "dbVol-"            
  Recycle Oldest Volume = yes
  Next pool = "europeBackupPool"
  Storage = dbStorage
}


/* europeBackupPool = the destination pool of copy job */
Pool {
  Name = europeBackupPool
  Label Format = "dbBackup"
  Pool Type = Backup
  Recycle = yes                       
  AutoPrune = yes                     
  Volume Retention = 365 days         
  Maximum Volume Bytes = 5G          
  Maximum Volumes = 1000             
  Label Format = "euBKVol-"          
  Recycle Oldest Volume = yes
  Storage = europeStorage
}

and this copy job

Job {
    Name = backupToEurope
    Type = Copy
    Pool = dbPool
    Selection Type = PoolUncopiedJobs
    Schedule = "dailyFullSchedule"
    Client = "none"
    FileSet = "none"
    Messages = Standard
}

Pay attention to the "Next Pool" directive from the initial pool as this is specifying the pool where the jobs will be copied.
After that when I run backup I get:

    Run Copy job
JobName:       backupToEurope
Bootstrap:     *None*
Client:        none
FileSet:       none
Pool:          dbPool (From Job resource)
NextPool:      europeBackupPool (From Job Pool's NextPool resource)
Read Storage:  dbStorage (From Pool resource)
Write Storage: europeStorage (From Job Pool's NextPool resource)
JobId:         *None*
When:          2015-04-01 18:44:27
Catalog:       MyCatalog
Priority:      10

The downside here is that if you want to restore this backup you would have to do it from the Asia Director, because the Europe director has no clue about this jobs...

Best Answer

Related Solutions

Bacula virtual backup job doesn’t run, no output

Bacula off-site backups

Related Topic