Archiving – what to do with all that footage?

Coming from a tape based Canon XH-A1 I found the digital workflow of the Sony EX1 to be a wonder with it’s instant clip reviews, no drop-outs, fast ingest and high quality. One problem with shooting on something like SxS, compact flash or SD cards though is that you have to put all of that data somewhere after each shoot. Gone are the days of storing the original tapes in a draw.

After around three years of using a digital workflow I’ve amassed a lot of data, my source clips alone use up around 6TB of hard drive space, add to that working files, transcoded scratch files, renders and all the other bits and bobs that get generated when working on a project and you start to need a lot of space on your computer system. Today I’m still shooting on the EX1, but the problem has only gotten worse with the addition of two DSLR’s, two GoPro’s and the Zoom h4n all filling up cards at silly rates.

Up until now I’ve simply been adding more hard-drive capacity to cope with demand, my mac is maxed out with four 2TB internal hard drives, a 2TB G-Raid scratch disk and an 8TB drobo which is struggling to keep everything backed up using time machine. My mac is once again nearing being full and with 2TB SATA drives still being the largest capacity available I really needed to come up with a way of offloading the older footage and freeing up my system for new files.

So let me talk about some of the options that I’ve looked into and then I’ll go on to describe what I’ve decided to do to tackle this ever growing problem.

Hard Drives
The good thing about hard drives is that they are one of the cheapest ways to store your files, with a 2TB drive costing around £100 that works out at approximately 0.5 pence a GB per drive.

Apparently hard drives have a tendency to seize up if not used for a few years and of course they are quite fragile devices. Drop a hard drive and all the data contained within it could be lost forever, unless of course you can afford to have it recovered by one of those super expensive clever people that can rebuild drives in a secret underground lab somewhere!

I’m a big believer in making sure that anything stored on hard drives is on at least two of them. Even if I’m using a drobo which mirrors the data across multiple drives I still make sure I have a backup on another drive outside the drobo. If a drive fails in the drobo then all is not lost, but what happens if the drobo fails and kills all of it’s drives? Call me paranoid but you can’t be too careful with anything that claims to look after your precious data.

Tape Backup
Tape systems such as LTO drives are designed with backups in mind, they provide a reasonably fast data transfer and the media is cheap. The drives themselves are quite expensive starting from around £1,000 and quickly heading into £2,500 plus for the most up to date systems. The tapes themselves offer one of the cheapest options for storing data, an LTO4 tape can store 800GB of uncompressed data and only costs around £25.

The downside with tape is that when you need to access the files you’ve backed up it can be a royal pain getting them back off of the tapes. I think as a backup system for use in data loss emergencies they do a great job, but for archiving they’re just not the right technology.

Sony Pro Disks
This was the technology I was most excited by when I started looking for an archiving solution, after all Sony must have thought about the workflow issues when they invented SxS and the Pro Disk seems like the answer. Unfortunately though it doesn’t appear to be ideal. Firstly the PDW-U1 drive costs a whopping £2,841. I’m not sure what technological goodness they packed into that little box but it must be extremely clever at that price. The disks themselves are around £45 for 50GB.

Sony Professional disks are used in the XDCAM range of cameras which record directly on to the disks allowing you to simply archive the actual disks at the end of a shoot and charge the media cost on to your client. I really like the thought of working that way but unfortunately if like me you’re shooting on an XDCAM-EX camera though you’d need to copy the footage from your SxS cards over to the pro disks at the end of every shoot.

I’ve been doing a little research into this and apparently it takes around 40 minutes to copy just 17GB of data. That doesn’t make it a practical solution at all. If I come back from a shoot with 60GB of footage on three SxS card it’s going to take three hours or more to transfer everything, and that’s with me being there to swap disks. That’s not going to work!

I know a lot of people backup to DVD and Blu-ray disks but personally I don’t trust re-writable DVD’s and BDR’s as far as I could throw them, which thinking about it might be quite far if using the Frisbee technique!. You get my meaning though. Some reports claim that the life span of these re-writable disks is less than three years if kept in ideal conditions and considerably less if not cared for properly. For me that’s just not reliable enough to be storing my footage on so I’m not even considering these at all.

So what’s the answer?
In all honesty I’m not sure there is a good answer at the moment. There really doesn’t seem to be a system that provides reliable long term offline storage with fast data transfer, low cost media and easy access to files. I’ve read that Sony are working on a ‘one time write’ system that uses 128GB optical disks with a long life-span but that’s no use to me until it gets released and there’s no mention of write times yet.

So for now I’m going to keep going the way I have been and use hard drives. Today I’ve blown £1,100 on another drobo, this time the drobo fs model packed with five 2TB drives which is going to become my new backup drive. I’ll use my existing 8TB drobo as a footage storage device. I know I have to archive some data so I’ve also come up with a plan for that.

Of the four 2TB drives I have in my mac, two of them contain footage that’s getting quite old and isn’t likely to be needed very oftem. My plan is to use a pair of 2TB external drives and copy the data from my internal drives onto the external drives. I’ll then pull out the internal drives and store those off site somewhere whilst keeping the external drives here in case I need access to the data at any point.

I know it’s not ideal to store drives without firing them up regularly, but it really does seem like the best option for now. Hopefully Sony or somebody else will address this problem soon, I’m sure I’m not the only person struggling to know what to do with all the data I’m accumulating.

let me know your thoughts and any solutions that work for you.


You may also like...

2 Responses

  1. Eelke says:

    Hi Paul,

    Struggling with the same stuff pruducing a lot of data, this blogpost is very useful to me. In fact, I’ll keep a close watch on your blog, because it gives a lot of good insights.

    I’m working on a good storage plan myself, and will let you know my final plan, if I’m satisfied with it.

    Keep up the nice work on the blog!

Leave a Reply

Your email address will not be published. Required fields are marked *