1010101010101

Archiving and backing up your projects

Info - Archiving and backing up your projects
Details

There's an aspect of post production which I've put off for a long time: backups and archiving projects. Mostly because every method out there came out to be a pretty expensive, or at least - more expensive then I'd like. External RAID arrays, NAS boxes, 19" rack-servers - I've been through them all, on paper at least. But I ended up with a slightly simpler solution which works great for me, and is very manageable.

Building a storage server can be quite expensive if you want to use a safe RAID setup. And with expensive, I mean that it can easily get into 4-digit numbers if you want to add a good amount of storage in a safe RAID configuration. And if you do it, you have to do it right, because it's hard and not without risk to increase the size of a RAID setup once it's been made. But that only gives you one copy. If something happens to your storage server (worst case: fire), it's all gone. So you need another copy of everything and things get more expensive. And by now things start to get more complicated. Too complicated for my needs, so this is not the turn I took.

Fantec QB-35US3R and HDD's

Backups

There's one thing you can be certain of when you store data on a harddrive: it won't last. It will eventually break down. It might be seven years from now, it might be the day after tomorrow. Even though some drives are covered by a five year warranty - that doesn't mean there's no chance it'll fail within the first. So backups are something not to be taken too lightly.

That being said. I'm personally increasing my risks by using a RAID-0 configuration for my current projects. It means that all data is being spread over all drives in the RAID. So if one drive fails, it's all gone. Backups become all the more important.

Earlier I mentioned that an external RAID array would become expensive quickly. But when you've got four matching terabyte harddrives spare (which I did, thanks to my newly built workstation) - its a cost easily overcome (though a 1TB drive isn't that expensive anymore, a good 1TB Western Digital drive costs a mere €55 (Amazon.com, Amazon.de, bol.com)). I've put them into a relatively cheap external RAID box, the Fantec QB-35US3R (Amazon.de, Amazon.com), and set it up as a RAID3 setup.

After working on a project I sync my internal RAID-0 with the external RAID-3 by simply double clicking one icon on my desktop. This is done at least once a day, so in case the internal array is lost, there'll never be more delay than 1 day. 

Even though I opted for this external RAID solution, it could also be easily done on two mirrored external HDD's. I wouldn't backup on an internal set - because some of the major causes of HDD failure would be applicable to the original drives as well as the backup drives (power supply failure (power peaking) and perhaps a failing chipset on the controller).

HDD Dock and HDD's

Archiving projects

For me it was important to keep things simple and manageable and still keep data safe. I got my hands on a SATA QuickPort Duo USB3 HDD dock from the German brand Sharkoon and a set of seperate harddrives (bol.com, amazon.de). For safekeeping I put them in Sharkoon's Quick Protects, which are nothing more than rubber sleeves, but it feels just that bit safer.

When I've finished a project, it is moved from the internal project drives to an archival harddrive (A). A second harddrive (B) is then inserted into the second slot of the dock. I then use a piece of open source software called FreeFileSync to synchronise/duplicate the harddrives. I intentionally don't use any real form of RAID for these drives because if one fails, the other will work on any system, using any controller and can be duplicated to any type/brand of harddrive if needed - something that won't always work with RAID drives.

Once the drives are synchronised - A is stored at one location, and B is stored at a different location. This will ensure data protection in case of fire or any other twist of faith. I might be in trouble though, if an earthquake would tear apart my hometown, but statistics point out that it might be a risk worth taking...

The drives are stored in a pluck-foam padded case, these are readily available from manufacturers like Pelican Cases.

Before I started shooting with my Blackmagic Design Pocket Cinema Camera (BMPCC) I used bigger 3.5 inch desktop drives for archival purposes. But now the data needed to be archived has increased in such a tremendous way that I've switched to 2.5 inch notebook drives in small cases. These aren't as fast, but they're smaller and cheap. Thanks to the data amounts which are a result of shooting CinemaDNG, only one or two projects would fit on a ~500GB drive, so the risk is more spread out.

The advantage of this archival method is that it easily expandable when the requirements are there, and the costs can be easily brought down to a per-project price.

 

 

The bits image used at the top of this post is made by Carsten Müller.

If you found this interesting, you might also want to take a look at these posts

Comments

Others have left 2 comments so far. Tap/click a comment to reply to that comment.

Paolo
Website for Paolo
Hi, I have the same problem.
Hi have researched a lot and my set up is this.
I copy all of my file twice, splitting them across two set of hard disks: originals and working copies.
I use an Icy Dock enclosure to add 4 external disks to my pc.
I then will subscribe to mega.co.nz for online storage and archiving. It is limited to 4 terabytes for now but I hope it will get better.
Take an eye on BTRFS, a linux file system that will solve many problems for us when it will be ready.
Ruben
Website for Ruben
Thanks for the BTRFS hint - heard about it once or twice, but I never looked into it.

Though it seems interesting, I'd like to keep my archive outside of a running system - at least for now. Online/cloud backups seem like a viable solution for work-in-progress projects as soon as a fiber internet connection with fast upload is available at the workstation-end. Though I'd probably prefer to use sattelite units with Bittorrent Sync for this purpose. I'm thinking of a low-powered Raspberry Pi with perhaps one big-ass drive attached. Of course two of these units on two seperate locations would do best.

Leave a comment

Name, e-mail and an actual comment are required. Website is optional. Your e-mail address will never be published.

Your comments has not been submitted succesfully. There is reason to believe it was submitted by a spam-bot. Please try again if you wish.

You are replying to a comment by [name]