Backup – Bacula GFS-Scheme Disk Backup Issue

backupbaculagfshard drive

I have a question about implementing the GFS-scheme with bacula on a harddrive(the deamons already work fine).

What I want to accomplish:

Every half year i want a full-backup that should be accessable for at least 12months ->
i need to keep the backups a little bit longer around than 12months that the second full backup doesn't overwrite the first one. That means there will be at least 3 full-backups around(I have chosen 18months). It continues like this for days, months and so on. Simple GFS-scheme.

Schedule{
  Name = "GGFSCycle"
  Enabled = yes

  # DailyCycle - the volume is kept for 14 days
  Run = Level=Incremental Pool=DailyPool FullPool=HalfAnnualPool Priority=10 tuesday-sunday at 03:00

  # WeeklyCycle - the volume is kept for 5 weeks
  Run = Level=Incremental Pool=WeeklyPool FullPool=HalfAnnualPool Priority=11 monday at 03:00
  
  # MonthlyCycle - the volume is kept for 7 months
  Run = Level=Differential Pool=MonthlyPool FullPool=HalfAnnualPool Priority=12 february-june 1 at 03:00
  Run = Level=Differential Pool=MonthlyPool FullPool=HalfAnnualPool Priority=12 august-december 1 at 03:00

  # HalfAnnualCycle - the data is kept for 18 months
  Run = Level=Full Pool=HalfAnnualPool Priority=13 january 1 at 03:00
  Run = Level=Full Pool=HalfAnnualPool Priority=13 july 1 at 03:00

}

Pool {    
  Name = DailyPool
  Pool Type = Backup
  Storage = backup-server-sd
  Label Format = "DailyVol_"
  Volume Use Duration = 23 hours
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 14 days         
  Maximum Volume Bytes = 50M          # Limit Volume size to something reasonable
  Maximum Volumes = 1000              # Limit number of Volumes in Pool
  # -> max 50G
}

Pool {
  Name = WeeklyPool
  Pool Type = Backup
  Storage = backup-server-sd
  Label Format = "WeeklyVol_"
  Volume Use Duration = 23 hours
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 5 weeks
  Maximum Volume Bytes = 500M 
  Maximum Volumes = 80
  # -> max 70G
}

Pool {
  Name = MonthlyPool
  Pool Type = Backup
  Storage = backup-server-sd
  Label Format = "MonthlyVol_"
  Volume Use Duration = 2 days
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 7 months
  Maximum Volume Bytes = 5G
  Maximum Volumes = 80
  # -> max 170G
}
 
Pool {
  Name = HalfAnnualPool
  Pool Type = Backup
  Label Format = "HalfannualVol_"
  Storage = backup-server-sd
  Volume Use Duration = 3 days
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 18 months #1,5 years
  Maximum Volume Bytes = 35G
  Maximum Volumes = 20
  # -> max 700G
}

The Problem I see:

is that an incremental backup always references the last incremental-, differential- or full-backup and saves only the files that changed since then.
In my scheme i have two "different" incremental-backups, the daily and the weekly one.
The weekly one always references the daily one, that came the day before, right? When i keep the weekly one around for more than 4weeks(i have chosen 5weeks) and the daily one only for more than 10 days(i have chosen 14days) the daily files are going to be deleted after 14days but the monthly one still references them. So there will be a "hole"?

I figured out, if it would be possible to have multiple "levels" of incremental-backups, like i could say that the weekly incremental-backup references only the weekly backups(level0) and the daily incremental-backups only the daily ones(level1) the problem would be gone.
The same problem would occur when i would keep the full-backup for to short around. For example if the full backup overwrites the only existing one there will be no full backup left that the differential-backup could reference and I lose all backups i made to that point.

What I found:

I looked around the internet and found a chapter in the bacula-documentation:
It states:"Now since each of these different kinds of saves needs to remain valid for differing periods, the simplest way to do this (and possibly the only) is to have a separate Pool for each backup type.".

My Questions:

That basically means that there is no other solution than the single full-,diff-, and inkr-pools in the chapter of the bacula-documentation. Is this correct?
If it's incorrect, how can i achieve my goal with two different incremental-pools?

I've also read about base jobs in the bacula-documentation, that's not the right thing but it goes in the right direction.

Thank you very much for your time and help reading that far.

Ps: I'm no native english speaker and I'm sorry if i do make mistakes. If there's something that really pisses you of I would like to hear from you to improve my english.

Thank You.

Best Answer

I've sorted it out with the help of the bacula-user-mailinglist by now. It is not possible to effectively have multiple incremental pools for a single job around. I've changed my settings to this:

Schedule{
  Name = "GFSCycle"
  Enabled = yes

  # DailyCycle - the volumes are kept for 40days
  Run = Level=Incremental Pool=DailyPool FullPool=HalfAnnualPool Priority=10 daily at 03:00
  
  # MonthlyCycle - the volumes are kept for 7months
  Run = Level=Differential Pool=MonthlyPool FullPool=HalfAnnualPool Priority=11 february-june 1 at 03:00
  Run = Level=Differential Pool=MonthlyPool FullPool=HalfAnnualPool Priority=11 august-december 1 at 03:00

  # HalfAnnualCycle - the volumes are kept 12months
  Run = Level=Full Pool=HalfAnnualPool Priority=12 january 1 at 03:00
  Run = Level=Full Pool=HalfAnnualPool Priority=12 july 1 at 03:00


Pool {    
  Name = DailyPool
  Pool Type = Backup
  Storage = backup-server-sd
  Label Format = "DailyVol_"
  Volume Use Duration = 23 hours
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 40 days
  Maximum Volume Bytes = 50M          # Limit Volume size to something reasonable
  Maximum Volumes = 3000              # Limit number of Volumes in Pool
  # -> max 150G
}

Pool {
  Name = MonthlyPool
  Pool Type = Backup
  Storage = backup-server-sd
  Label Format = "MonthlyVol_"
  Volume Use Duration = 2 days
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 7 months
  Maximum Volume Bytes = 5G
  Maximum Volumes = 80
  # -> max 170G
}
 
Pool {
  Name = HalfAnnualPool
  Pool Type = Backup
  Label Format = "HalfannualVol_"
  Storage = backup-server-sd
  Volume Use Duration = 3 days
  Recycle = yes
  AutoPrune = yes
  Volume Retention = 12 months
  Maximum Volume Bytes = 35G
  Maximum Volumes = 20
  # -> max 700G 
}

file- and job-retention for that single client are set to the maximum volume-retention.

-> File Retention = 12 months , Job Retention = 12 months

There are also Virtual Full-backups which may come handy and are explained in more detail at: https://www.baculasystems.com/incremental-backup-software/

Once again Thank you very much to everyone on the bacula-user-mailinglist who helped me figure it out.

Related Solutions

Bacula — splitting volumes

In order to get Bacula to do what you want you need more than just volumes - you need to put those volumes into separate pools and let your jobs know that you want to use different pools for different backup levels.

The magic syntax is:

Job {
        Name = "Test"
        Type = Backup
        Client = backup-fd 
        FileSet = "FileSetTest"
        Storage = SomeStorage
        Schedule = "ScheduleTest"
        Pool = Default
        Full Backup Pool = FullTest 
        Incremental Backup Pool = IncrTest
        Differential Backup Pool = DiffTest
}

(shamelessly thieved from http://wiki.bacula.org/doku.php?id=sample_configs -- Check the bacula docs as I think there are probably other places you can specify the per-level pools, like JobDefs & possibly Client)

You would then set retention periods on the pool (or constituent volumes) to meet the requirements you outlined earlier.

On the subject of disk space, I've found it's better with disk backups to treat them like they're tapes.
I suggest you define a "reasonable" `Maximum Volume Bytes in your Pool resource (and update any existing volumes to reflect it), and then create a bunch of volumes that Bacula will cycle through on its own as it "fills" each volume, and recycle in accordance with whatever retention policies you've set up.
In my case I have 200G of space for backups, broken out as 100 2GB files.

This has a few advantages:

The backup volumes fit on a DVD
(so if I need to archive them forever I can just chuck them onto a disc)
Off-Site sync only needs to push the files that changed
(smaller than one 200G volume)
Faster disaster recovery
(If I ever need to recover a server using my remote archive I only need to download its bootstrap (.bsr) and the volumes I need to restore it.)
If my hard drive dies I have some hope that it will only kill some of the files.

Setting up a bacula file-based volume retention policy

Bacula creates volumes automatically up to the 'Maximum Volumes'. But you can always create a volume manually by using the bacula console. According to your configuration, your 'Maximum Volumes' appears to be set to 2, 10 and 7.

Depending on your backup strategy, you may need more volumes for the full backup and less for the incremental and differential. I would suggest to increase the number of volumes for your SVNFullPool. This depends on your content and the maximum size you defined on each volume.

All this also depends on your backup schedule. If you do a full backup several times during a retention period, you have to set enough volumes to hold several full backups.

The following article gives some feedback about pool definition, schedule and it provides interesting statistics about the number of volumes that were used for a one year backup.

http://blog.vacs.fr/vacs/blogs/post.html?post=2010/04/09/One-year-of-data-backup-with-Bacula-on-a-ReadyNAS-duo

Best Answer

Related Solutions

Bacula — splitting volumes

Setting up a bacula file-based volume retention policy

Related Topic