ASK & DISCUSS
INDEXTIP - Amazon AWS and S3 as footage backup
4 years, 11 months ago - Marlom Tander
Just a heads up, but if having your raw footage on cards and drives keeps you up at night, never quite sure that none are lost, corrupted, and that the house might burn down....
Using Amazon AWS and their S3 storage is a really useful low cost backup.
Set up an account, a bucket, and upload as fast as your upstream connection will allow (much faster than FTP, I've done 10GB an hour upstream on my home BB, and I'd guess it might be even faster if left overnight so people aren't hammering Netflix...).
You can create additional account users, and so if you have a lot of data each day, share the drives out between 2, 5, 10 whatever people and have them all upload their share via their BB overnight.
For longer term storage (if needed, e.g. until post is complete), you can move to the Glacial storage, which is cheap as chips, but isn't instant access.
Costs are modest (basically entirely activity/volume driven), and once it's there, you know it's safe from physical danger. There is a FREE activity level which is fine for getting to grips with the system.
NOTE - it is NOT a "yea, upload it and let the editors use it as an online hard drive" because that way it gets expensive, fast.
Only members can post or respond to topics. LOGIN
Not a member of SP? JOIN or FIND OUT MORE
4 years, 11 months ago - Ian Allardyce
Hi marlom, good shout. I'll be producing a feature doc later this year and next, so might look into this. thanks
Response from 4 years, 11 months ago - Ian Allardyce SHOW
4 years, 11 months ago - Paddy Robinson-Griffin
Cheers for the update Marlom!
It's amazing that broadband connections are fast enough to do this these days! I looked into it some years back, and it was going to be faster and cheaper back then to mail them a hard drive in the post and pay for ingestion than try to upload all the material!
S3 is redundant storage, too, so multiple copies are kept. There's savings to be had from reduced redundancy, but I wouldn't suggest it.
As you note it's not meant for a lot of up/down traffic - if anyone did want that, they should consider breaking big files up into the smallest possible parts to reduce overhead. It's also not optimised as a streaming server - this can be added as "Cloudfront" which is also great value, but effectively temporary local copies to the streamer. For most ohmygodohmygodohmygod backup circumstances, Glacier (the extra slow service) is ideal.
In each case, make sure it's not your only copy and read the SLA. If they lose your movie, I'll bet they won't cover the reshoot! Insurance normally dictates 3 geographically separated copies on hard drive for safety, this would probably count as an extra one for safety and convenience, check your policy to see if they'd accept it as one of the three :)
Response from 4 years, 11 months ago - Paddy Robinson-Griffin SHOW
4 years, 11 months ago - Marlom Tander
Amazon take NO RESPONSIBILITY for the data. AWS is big boys territory, you are the God of your account, and if you click "Delete Bucket", it's not in a trash can :-)
It's only recently that they defaulted new Buckets to Private. (Those huge data leaks you hear about, mostly companies AWS systems, wrongly configured.)
But, I view S3 as an extra backup to whatever is in place already in hardware.
Response from 4 years, 11 months ago - Marlom Tander SHOW
4 years, 11 months ago - Paddy Robinson-Griffin
@Marlom Tander Good call :)
Response from 4 years, 11 months ago - Paddy Robinson-Griffin SHOW
4 years, 11 months ago - Marlom Tander
I hate having mission critical on hard drives. There is always that niggle, even if multiple copies. Mind you, I did my first programming hand punching cards, when "the student delegated to bike them the 4 miles to the mainframe got rained on and the cards got wet and, you'll need to do it all over again" was an actual thing :-)
Response from 4 years, 11 months ago - Marlom Tander SHOW
4 years, 10 months ago - Marlom Tander
Update - Wasabi seems a lot cheaper than AWS S3, as a pure storage option. Flat rates at $6pm per TB. Not yet experimented with it though.
Response from 4 years, 10 months ago - Marlom Tander SHOW
4 years, 10 months ago - Paddy Robinson-Griffin
Reminds me of something else worth mentioning, if your video is stored in S3, you can transcode it fast using Amazon's servers. Stroes it back in S3, and frankly you can then CDN it out with Cloudfront.
Response from 4 years, 10 months ago - Paddy Robinson-Griffin SHOW
4 years, 10 months ago - John Lubran
We have hundreds of hours of our own archived footage on most tape formats since the early 80's. So far even the oldest ones seem to be in good order.
Sadly the mooted 4K tape format hasn't yet been able to persuade producers that possessing robust hard copy in oned own cupboards trumps the convenience of hard drives that are unlikely to be as robust as our quarter century old tapes. Mind you, keeping VCR's able to play those tapes is another issue.
Thanks to Paddy and other generous donors we are able to keep some of these valuable machines going.
Response from 4 years, 10 months ago - John Lubran SHOW
4 years, 10 months ago - Marlom Tander
UPDATE on Wasabi
Their low low prices are based on your monthly download volume not exceeding (by too much), your storage volume.
So, really good for backup purposes during production and post. I.e. my use case.
Not so good for anyone thinking of setting up a video content site.
Response from 4 years, 10 months ago - Marlom Tander SHOW