Cost Effective Long Term Archival of Video and Image Data – 50 TB Solutions

archivebackupstorage

My lab is in the process of setting up a small server that holds data (mostly video and image data, plus a few documents) for the project our group is working on at a moment in time.
Historically, after a research project ends, the data haphazardly ends up being archived in one hard drive, or a big pile of DVDs (or CDs in the olden days), and/or some of the video ended up in Sony DV cassettes or even VHS tapes (this lab has been active since the early '90s), OR a mixture of all the above…

Question: What is the best way for (1) consolidating them ALL into the same format AND storage medium, and (2) what's the best medium for long term archiving of such data for very occasional access (say, 30+ years?)? Unfortunately we don't have enterprise level budget (we are just a ~10 people lab), so can't do things that costs hundreds of thousands of dollars.

Thanks!

P.S. Considering our old video and images are of smaller resolution, but recent ones are huge, I think we are talking about 30~40 TB for the really old data, another 10~20 TB for recent data, then yearly additions of about 5 TB.

Best Answer

Unfortunately, there is no best way for you. 30 year archival of digital media is a very hard problem and takes routine investment. About the only formats guaranteed to be readable in 30 years are ASCII and UTF8, which are not video formats. Storage formats change, the 8 track reel-to-reel tapes we were using 30 years ago are nigh impossible to read these days even though the data is still on the tape (there is an interesting story about NASA rebuilding a 40 year old tape drive to get at some newly recovered/discovered Apollo data tapes). Your best bet is to commit to periodic, I'd say every 5 years, assessments of your archival environment with sufficient budget to bring old formats into newer formats.

You probably know better than I do, but the video landscape is changing rapidly. Realtime online editing is now possible, where it was only doable on seriously good kit even 10 years ago. Who knows how things will look 30 years hence.

  • Set your archival window for 5 years.
    • In the immediate term a largish storage array should suffice (
      • big and slow 50TB disk can be had for under $70K, possibly well under.
      • An LTO5 tape drive and 50 tapes (well over 50TB worth) can be had for less than $15K.
  • What format you store your video in is up to you.
  • Start finding and converting all of your older stuff into this new storage.
  • At the end of 5 years, do another full assessment of your archival environment.
    • What formats are you using?
    • What are newer formats?
    • What codecs seem to be dead ends, and what media do you have stored encoded that way?
    • Decide how you're going to migrate to newer storage methods (data formats, disk/tape/something-else), and spend appropriately.
  • Repeat 6 times.

That should get you to 30 years.

Related Topic