ASK & DISCUSS INDEX

11 years, 11 months ago - conrad gaunt

could I ask what camera it is that will be generating that load? also, how many cameras? 24hr timelapses? im struggling to get my head around those data rates..

storing on harddisks will not be cheap, and you really need two sets!.. LTO is not cheap..

150mb (not terrabytes) is the average for a red one 's raw for a day.. 400mb ample on an epic if not shooting 120fps 5k all day.. I guess my answer would be, if at all possible, shoot as efficiently as possible and generate less data . we could rent you a red one or an epic camera that shoots raw for less than any other camera you could rent..and wouldn't need 1.5tb a day!!!! If shooting blackmagic raw (uncompressed), or alexa raw, you should compress it for storage (raw jpeg2000 approx. 60% original size) ..or use a compressed option..or just rent a camera package from us that does compressed raw and save yourself £k++

red one kit £1200 month (150-250mb a day)
epic kit £2k p/month (350mb a day)

k.r
Conrad gaunt
north Yorkshire camera hire limited
07835915253

11 years, 11 months ago - Dan Selakovich

This is way off topic, but the director is editing his own picture? That's a mistake. Even if he's a professional editor. In short, a director doesn't have the objectivity to do the best job on his own work. Please reconsider this.

11 years, 11 months ago - Miriam Rodriguez

Hi all, and thank you for your replies so far.

To answer: this question was sparked by a conversation with a DoP with an F55, who suggested that, if we were doing some minor work in After Effects, it would be best to work with the RAW files. He then estimated that it would require around 1.5TB of storage per day to do this.

This is not a documentary and we will be limiting the amount of footage to what is necessary. The director is also the editor (on a Mac Pro – not a laptop) and we are practiced in doing this with short films, however, although we have shot on Red One before, we haven't yet used the F55 and so were going under the advisement of the DoP. That being said, the DoP is now unavailable for this shoot, but we still may end up with an F55 or equivalent, so the question may still stand.

You are quite right if you have mentioned that it's the producer and director's job to ensure that a budget is allocated wisely and "evenly" across the film's production (and nothing makes one more eager to scrutinise expenditure than self-funding a feature!)

This is something that we will again discuss with the DoP that we eventually end up working with, and we would be very happy to learn that 1.5TB is a wild overestimation.

For now, we would still be happy to hear your opinions on the safest and most cost-effective ways to store and backup the footage.

Thanks again!

Miriam

11 years, 11 months ago - Michael Patrick Kelly

Yeah... 40 TB sounds wrong. You're probably looking at no more than 4 TB and that's shooting quite a bit of footage. I can't imagine filling 4 TB in 12 days... Our latest feature was shot HD 4.4.4.4. and we filmed 150 scenes over 25 days and we did not fill 4 TB drives.
The important thing to remember is to have your DIT person back everything up to at least one duplicate set of drives.
We had AB&C drives - two sets of backup. Good luck!

11 years, 11 months ago - J. D.

Look for an LTO system, they are great for archiving footage. http://www.lto.org/ LTO6 is the latest greatest, but I think you can find some used systems, if budget is a concern.

JD
www.cineshooter.com

11 years, 11 months ago - Peter Ward

Sony. 4K 16bit linear RAW is 432GB/hour. As raw codecs go this is pretty compressed. But what to really know what storage solution is best we need to know the intended workflow. Are we talking all 40GBs or raw online and ready to edit? Of small pieces, the rest of the footage archived meanwhile?

The safest way to store video, or any other data, is lots of copies. No single drive or RAID should ever be trusted. Internal drives or a Drobo are great for the archive. You'll need someting a lot more serious when actually working with the RAW; which specifc model depends on variables: amount of footage, number of tracks. the model of the host computer and possibly the OS. Most likley a tower with x8 pcie will be required for a SAS RAiD card; thunderbot may even be too slow.

-Peter

11 years, 11 months ago - Sandy Chase

How about Drobo 5D? 5-bay RAID that takes 5 x 4TB drives like the WD caviar black 7200rpm enterprise drives. RAID 5 protects against single drive failure. Daisy chain multiple arrays through thunderbolt (though not on current Mac Pros.) And you can slip in an SSD in one of the bays that it uses intelligently for better performance.

11 years, 11 months ago - Paddy Robinson-Griffin

I know I've already said my piece, but I'd just like to expand a little on my motivation for my thinking - it may help or hinder and by all means feel free to disagree, just wanted to offer.

Production is all about balancing costs. As production, it is our job to look at the numbers and say to the director/other HoD "that's nice dear, but you can't afford it". When the director insists that the helicopter shot is essential, it is our job to ask him which scenes he wants to cut from the script to pay for it. We do the same between departments. There's no point shooting on an iPhone but then grading on a Quantel. There's no point having great costumes if the lighting is lousy. There is no point having a great location if the audio isn't crisp. We have to balance between departments, and sometimes that means saying 'no'.

If you've got a 'micro' budget, you need to balance very carefully. One department being way out of sync with the others just damages the overall project, the extra money/time/resources could be spent to higher effect bringing another department up to spec. It also sets you up for more waste or extra costs - for instance if you have 4k RAW images but grade them on a laptop (as opposed to a proper suite and real calibrated projection) you're wasting the extra quality you paid dearly for. That, or you spend another micro feature budget's worth on doing a proper colouring alone. Then the sound will let it down, so you ADR/Foley the lot. Now the score sounds too thin, so you need it rerecording... you get the idea.

My 2p

11 years, 11 months ago - Tommy Boulding

To get more specific about storage; the three features I have cut this year have all had their data stored on G-Raid external hard drives (either 4tb or 6tb). The DIT has had a thunderbolt card reader and backed up to two identical drives using either Al3xa Data Manager or similar. One of these drives has been taken off site every night while the other has been left transcoding the dailies to Proxies overnight. The proxies have been stored on a separate external 2tb G-Raid and this has become my working edit drive with all sound rushes, project files, music etc stored on it so I can take my project wherever I want and know all I have to do is plug the drive into a new suite and it will all be there.

For what it's worth I have also been making automated backups of my project files to Dropbox.

Hope that helps.

11 years, 11 months ago - Sandy Chase

Nothing exists unless it exists in at least two places. Preferably one of those off-site. I am assuming that you're not going to use these on-set backup drives to edit RAW footage directly. You should edit with proxies and then "online" the footage later in the post process. If you edit directly with the RAW footage, you're going to need a monstrously fast RAID array to handle that.

11 years, 11 months ago - Timothy Wood

If your looking for an external hard drive solution I'd buy 2 x 5 bay raid enclosures and 10 x 4TB hard drives it comes to £1654 by my calculations.

I was looking at similar problems my self as I own a sony that can use the 4k recorder and the RAW recorder uses ALLOT of space. Are you shooting using 4k RAW on A sony? I'd be happy to help DIT if you need it and if you find a better solution let me know.

11 years, 11 months ago - ANDY LEWIS

I believe in the film industry, on set, they would only print selected takes. The editor would not have to wade through useless shots. I edit my own films. I know it's bad, i wish I had an editor, but it's micro budget. Also , I usually edit with the last take. That's because I directed it. I get a good take I move on. I don't have time to make notes on set of which take is good, or the budget to have another person with a note book. If your shooting ratio is 10:1 there must be scope for NOT SAVING all your footage. It's too easy to drag the whole camera file onto the hard drive. I know, I do it myself onto my 2TB drives. I have four WD's. No failures with any of them. Has anyone tried their 6TB thunderbolts? I could maybe buy one or get another person on set.

11 years, 11 months ago - Connor Snedecor

1.5 TB a day means your DOP expects to be shooting three hours of footage a day, assuming you're shooting 2.5K raw on a Blackmagic. They claim 30 minutes of raw footage will fit at 24p on a 256 gig drive. So... is this a documentary? If not, 3 hours a day sounds way out of line with a micro budget feature unless you're running multiple cameras constantly and shooting documentary-style (lots of improv, etc.). If it's being shot single camera and has a script, you shouldn't be coming anywhere near that.

The last short film I made we shot 9 400-foot rolls of 16mm film, which works out to 99 minutes, in 4 days shooting. Obviously we were trying to be economical and the film ended up around 9 minutes long, so 11:1 shooting ratio. But unless you're doing observational documentary, the camera should not be rolling so much each day. You should be aiming for a ratio in that ballpark. 12 days * 3 hours = 36 hours of footage total for a 90 minute feature = 24:1 shooting ratio.

So yeah, listen to other people about shooting economically and talking to your DOP about this math.

11 years, 11 months ago - Nick Currey

A few small points. I agree that the DP might need a talking to. However, some people seem to think the 40TB is all the original footage, but as Miriam says that's how much she calculates it will be including a back-up; so 1.5TB x 12 days = 18TB, which is still massive but less scary.
For editing the primary material (however large that turns out to be) they'll want to use a fast interface like Thunderbolt and those drives are not cheap. However, the back-up can be stored on anything you like - it's a back up so won't need to be accessed directly. You can get a 3TB drive that uses bog standard USB2 for about £70 on Amazon nowadays (how things change), so it needn't break all the banks at once. The transferal of data will be slow, but just set it going overnight.
But setting aside the 'weight' of the footage, let's talk 'length'. I heartily agree with Tim's point that we editors are increasingly being presented - confronted - with DAYS of digital footage. I know that Reality tv use armies of editors and support staff to deal with the massive amount of material - Ok, that's in the nature of the form. But many of us are lone editors working very quickly for a small client on a micro budget. It's not unusual nowadays for me to be given 15 hours of material for a project that will end up as a 5 minute online doc and a v. quick turnaround. Slightly longer form projects can have 100's of hours of footage. Editors always used to view all the footage before starting an edit. Presented with a tsunami of material we no longer have this luxury. There is literally not enough hours in the day.

11 years, 11 months ago - Dan Selakovich

Listen to Paddy. I couldn't agree more, especially his 2nd post! When I first read this, I couldn't quite understand your shooting ratio. People who shoot digitally tend to ignore this for some reason. In the film world, it's not just the cost of film that sets the shooting ratio, it's the schedule and a dozen other factors. Your ratio seems way up there. In 12 days, I think a 6:1 ratio on a single camera shoot to be pushing the limits of your crew, not to mention that there are only 24 hours in a day. Possibly if you were shooting it all on one location with multiple cameras, you could push that ratio to 12:1 in 12 days. Please keep in mind, that DPs are concerned about one thing. They will, in general, push sound into a corner if it means crap sound, but a good image. As an editor, one thing I always do on every show is ask the people that come after me what they expect. So, in my case, that means, the lab (when needed) color timing, the mix, ADR, negative cutter (when needed), the transfer to 35mm (when needed), etc. In the film world, much of the answers I would get are answers I expect, but I still asked. In the digital world, there is not much standardization, so these meetings with people in the following steps are increasingly important. Don't assume anything. As Paddy says, color timing on a mac laptop really means you are probably wasting your money on 4k footage. Money that could be better spent in other departments.

11 years, 11 months ago - Marlom Tander

On the editing side, I certainly think that directors should shoot as if they were editing it. And if they can't, they should at least do a first filter on the material to make sure they pass the editor a managable amount. I hate editing, but wearing a community media hat for a few years, had to edit my own material. I became quite good at shooting no more than I needed :-)

11 years, 11 months ago - Tom Hruby

What a boat load of fantastic info on this thread, tremendous experiences to gather this kind of info. But, seriously makes me want to run fast back to tape format, tiny little storage compared to this. You say micro budget, I've done shoe-string features - final editing one right now, this sounds to be going overboard before filming even gets off the ground. Point it, shoot it, move it (great planning required). Tech won't bring out the brilliance, neither will excessive shooting. Love the note about Skyfall, if that doesn't twist your thinking nothing will. Truly best of luck with your film.

11 years, 11 months ago - Suzanne Smith

Your DOP is right to overestimate- even if you don't accumulate that, it's better to be prepared and have a system in place and funds available to deal with what could happen. The best way to store a massive amount of data is on magnetic tape, this is a standard way that IT companies do it. I wouldnt store my main rushes like this, only the backup. Your DIT should be able to advise you on this. If they can't then get in touch with some edit houses in Soho and ask around. Usually there would be a relay back to the edit house on a daily/weekly basis, so you don't actually have to deal with all the data on location. I hope that helps and good luck!

11 years, 11 months ago - Paddy Robinson-Griffin

Holy crap - micro-budget feature and you're looking at 40TB storage?! It won't be cheap. Satellite maps of the planet earth are about 8TB total, it's a heck of a lot of data, massive. Bear in mind drives have a failure rate of approx 6%/year, so if you don't plan on risking losing at least some of your content, you need to duplicate that - so store 80TB. Online storage, even at the cheapest AWS/S3 glacier rates is ~$1000/month, if you want to actually be able to use the files, maybe ~8000/month plus bandwidth charges, so allow $10k/month. Clearly way out of proportion. Shooting on 35mm would be WAY cheaper than this sounds, this isn't the digital revolution in action.

Do you really need to keep RAW?? I mean, there are some excellent codecs out there that address just this insane problem. Shoot to DNxHD or Pro-Res. Seriously, unless you're planning on grading in a specialist facility and dropping another £100k+ on colouring, you're just going to make expensive problems for yourself. Do you know the difference between 8 and 10-bit per channel? Can you even see the difference between 4:4:4 and 4:2:2? I would seriously consider going back to your DoP and suggesting they get a grip on a micro-budget feature, not a Spielberg epic. Possibly your DoP is inexperienced and hasn't really thought about this, so just decided to use the most bestest format he can think of, but it is so inappropriate to my mind. As a producer/PM I urge you to push back.

11 years, 11 months ago - Paddy Robinson-Griffin

Technically, yes for AE work, RAW is preferable...BUT it is pretty marginal and the usual workflow is to shoot to a compressed format and then convert the few shots you're trying to get funky with to a bitmapped string of images (effectively same as RAW), do the FX work, and recompress as required. Needing a few moments of uncompressed video is a dreadful reason to shoot 40TB! From a production point of view, probably not a great loss if you're using a different camera HoD now.

Best way to store a TB or under? Go to Maplin/PCWorld and buy some external hard drives. You can even get mirrored pair enclosures, but storing the drives separately reduces overall risk, just make sure your DIT keeps them in sync.

Mac Pro/laptop either way it's a consumer grade monitor I imagine (if it's under £10k it's likely to be), and will only ever give you a *guide* as to the grade. Same with AE, BTW. It's an alien concept to many people, but different monitors have different colour profiles. Some will be a bit greener in the shadows, or bluer in some mid-highlights. If your colouring looks perfect on one monitor it might ming on another. Or if not ming, just be off-colour a bit, defying the point of the grade. This is why professional kit is highly tolerenced and properly calibrated, to give true reference colours. Another thing to consider is 4k - really? Remember the last Bond was shot on the Alexa at ~2k, are your demands actually higher than Skyfall? You won't have a 4k monitor (let alone a 4k reference monitor) to grade or do FX on, so again you might just be making trouble for yourself down the line.

I am not trying to cause you trouble here, it's your project and I am making a lot of guesses and assumptions. I just know there is a HUGE amount of fetishism in camera departments in film, especially at lower budgets. Inexperienced DoP's will swear blind that they absolutely must must must have utrabright primes, then spend more money on the best ND filters to stop them down and get narrow depth of field for that 'film look' (ie the look created by slow lenses 50+ years ago...). I spent a week being able to watch Janusz Kaminski and his team in action. A lot of the time they shot on multi-length lenses, not even primes. Yes, good multi-length lenses, but effectively zoom lenses that inexperienced cinematographers often eschew. Be very aware of camera department fetishism, it is always worth sanity checking what they insist is the ONLY way to work dahling. ;-)

11 years, 11 months ago - Tommy Boulding

I have to agree with Paddy. 40tb seems massively over estimated. I am currently cutting a feature which was shot on alexa on 2 cameras for a 24 day shoot and used about 12tb of data (so 24tb once it's duplicates for backups). All of it was shot at 2k and ProRes 4444. The B camera was running 80% of the shoot.

So unless your shooting documentary style I think the 40tb figure will need revising. You need to make sure the 1st AD is running a tight ship in terms of camera discipline. It's all too easy for them to call "roll camera" well ahead of when its actually required and therefore waste so much potential hard drive space. If it was film they wouldn't get away with it so don't let them just because its digital.

11 years, 11 months ago - Tim iloobia

One thing I wanted to contribute to this discussion from an editors point of view is I am increasingly getting vast, vast quantities of unrestrained rushes requiring huge amounts of storage space and requiring increasing time distilling them all.
I always try to have a dialogue with the production crew at the outset of a project if I am going to edit it to make sure we are all synched up and prepared for the work flow, and what that work flow is.
DOP's are wonderful, but it is the director /producer who ultimately should be in control of shooting ratios and quantity and QUALITY of footage. To shoot economically and with control is the challenge. To keep rolling rolling rolling demonstrates little restraint and soon becomes someone else's real problem further down the production line. And so much wastage!
Just because one can doesn't mean one has to shoot endlessly. The art of restraint seems to be something that gets forgotten more than it should. And there seems to be an over enthusiasm to jump onto the next ever emerging trend (lets shoot in RAW for example - it gives you more grading opportunities etc. well, yes, you will see the benefits if you have the money for a high end grading suite and imagine your film screening in a 4k cinema, but is you micro budget compatible with that?) The seduction and distraction of the current technology which will be yesterdays news next week when everyone wants to shoot in 8K doesn't always equate to making great films - it can cause such technical hellholes that one is in danger of losing sight of what one is doing. A micro budget should surely be an invite to innovate with the wonderful tools we have available to us but not create the workflow of a major, well financed production.
Your DOP probably means well but do you need to work like this, in reality?

11 years, 11 months ago - Peter Ward

Internal hard drives like HGST 4tB Deskstar and a dock will be the most cost-effective. Be sure everything is carefully labeled. And hopefully your DP is making proxy, otherwise having 40tb online you're probably talking SAN and hiring a networking tech.