Contribute
Register

Need help with the internal "best practice" storage concept using NLEs with my upcoming Hackintosh

Joined
Dec 26, 2016
Messages
123
Motherboard
Gigabyte GA-Z170X-Gaming 3
CPU
i7 6700K
Graphics
RX 480
Mac
MacBook, MacBook Pro
Classic Mac
Power Mac
Thanks, just read the same in the official Gigabyte manual (page 17). Really strange what they did there...
The 2nd m.2 connector works as I'd have expected (doesn't affect SATA ports all all when using PCIE blades).
Except that you'll immediately loose SATA Port0 also on the lower M2 port and only have 5 of 6 SATA ports left.
 

Fl0r!an

Moderator
Joined
Jul 6, 2010
Messages
2,714
Motherboard
Asus Z170M-Plus
CPU
Intel i5-6600K
Graphics
AMD Radeon R9 280
Mac
MacBook Pro, Mac Pro
Mobile Phone
iOS
Joined
Dec 26, 2016
Messages
123
Motherboard
Gigabyte GA-Z170X-Gaming 3
CPU
i7 6700K
Graphics
RX 480
Mac
MacBook, MacBook Pro
Classic Mac
Power Mac
For now i only thought using an AHCI M2 256GB SSD, for example Kingston Predator series for the OS+application drive.
The pricing is more expensive than an 512GB Samsung 800 EVO SATA drive.
The key factor is that the M2 drive has a almost twice that fast performance of the SATA drive with ~500MB/s
Plus i'll loose 1 of 6 SATA ports when i use (in best case the lower) M2 slot.
The question is... does this make sense and isn't a SATA SSD enough for a plain OS+application drive?

In an ideal world i'd also cut data from daily "all round" office work, downloads and so on from the OS+application drive to keep it clean, also for backup purposes. This is why i thought on a 512GB SATA SSD, or perhaps a HDD, or just a partition of a bigger HDD.

In this case there would be internal 2 SATA ports gone.

Then there's the NLE and Lightroom part where 4 SATA ports are left. SSDs are not ideal in every situation, but compared to HDDs they come handy when it comes down to power consumption, loudness and heat if you usually plan plenty of of them.
I think a combination of SATA SSDs and HDDs make sense but this is the part i'm beginning to struggle around.

In the upcoming months i plan with an external backup solution. I'm still not clear of the bus i want interconnect it to, but i don't plan to realize an internal backup strategy.
 
Joined
Oct 5, 2013
Messages
507
Mac
Classic Mac
Mobile Phone
Thank you for your thought.
I just started a new thread not to flood this one with my off topic store concepts headache.
Please feel welcome to add your knowledge here cause i already stated in detail my thoughts here: https://www.tonymacx86.com/threads/...h-my-upcoming-hackintosh.211785/#post-1409710
There are some things to keep in the back of your mind (along with all of the other stuff :) ) the lessons from older approaches still work:

Price is always a considerwtion, but the SSD are going to help you more than HDDs will because of how they work. Both are fast still fast enough for HD video and 4k video but HDDs slow way down with multilayer projects from miltiple source files because of how they access data on the drive, their advertised high-speed bandwdth drops dramatically. SSDs do not have that problem, but are still slightly more expensive -- if you get the enterprise class ones. The consumer ones are getting quite reliable and cost less. Note: with 8k editing staring folks in thhe face, HDDs may not be able to be as useful other than as archival types of storage.

If you can, keep the audio files on separate disks from the video files. Again cost is an issue, but your nle will not stutter.

Earlier I mentioned using external RAID with either eSATA or USB, you can also build them using a 10gb ethernet cards and get very fast connections. But stay with RAID 10 or maybe RAID 5 depending on your budget.
 
Joined
Dec 28, 2016
Messages
23
Motherboard
Gigabyte GA-Z170X-Gaming 5
CPU
Intel Core i5-6600K
Graphics
Gigabyte Radeon RX 460 OC
Mac
iMac, MacBook Pro
Mobile Phone
iOS
While I don't do that much video editing anymore I went with an 850 EVO SSD for my boot drive and a regular HDD for a data drive. I am planning on keeping the boot drive as clean as possible and offloading everything to the HDD that I don't need speed for. I also just picked up an HGST 7200k 3TB drive for my photo storage (1.5 TB and counting) because the other internal HDD is only 5400k. I personally decided I needed to stop using so many external drives.

I thought about going with an M2 but knew that I was going to lose some PCI and SATA lanes and decided that an M2 device would come later once I got the system stable. Also I'm pretty sure M2s were only supported on Hackintoshes relatively recently (I can't find the thread at the moment) so I went with an SSD. The good thing about building your own machine is that you can throw new parts in whenever you like!

I am going to reiterate jerryy2345 here: do not do striped RAIDs, please. They will only end in heartache. I agree with RAID 10 and RAID 5 though do RAID 6 if you can. Every single one of our SAN drive chassises have LUNs built with RAID 6. I have seen some pretty interesting drive failures that were survived because of RAID 6. You will lose some write speeds so for non SAN arrays I'd say RAID 10 or 5 will do you just fine.

You mentioned external backup solution. I would highly recommend building a NAS (or buying one if you don't feel like dealing with FreeNAS or one of the other ones) and either throwing it on your network over 1GbE or doing a direct connect using 10GbE. This would be a place to both have a living backup of your boot drive plus any important data and also a long term archive solution. We do have clients editing off small NAS solutions but the NAS has a minimum 10 GbE connection to the house network and I just installed one that has a 80 GbE (2 x 40 GbE) to the storage switch and a 20 GbE connection to the house network. So you can try editing on a NAS at 1GbE but ymmv.

I've been running a relatively small FreeNAS setup on my network for a while and I use it as a backup target. I used to use CrashPlan but I started to get annoyed with it (the agent is Java based) and decided to switch to Arq Backup (www.arqbackup.com) on the advice of a friend. I have Arq Backup sending to both Amazon Drive (60 bucks/year but free three month trial) and to my NAS. This way if I have catastrophic data loss I can restore from my NAS for stuff I need immediately and from Amazon Drive for less important things. Arq Backup is 50 bucks but you can do a month trial. It also encrypts everything.

Of course the bottleneck here is your internet connection! The initial backup will take a significant amount of time, I am dreading pushing 1.5+ TB to the cloud again but the peace of mind is worth it. I had an external drive fail on me and would have lost 85% of my photos if not for CrashPlan. Of course I hadn't built my NAS at this point so I had to recover from the cloud which took some time.

Anyways I'll get a trial of Premiere loaded on my machine and run that test I mentioned. I'll also run it in FCP X with the following variables: media on 5400K HDD and media on my 850 EVO. And Avid MC 8! Because why not, I learned it in school.

*edit* If you want an insane option go with direct attached storage over fibre for your media. Pick up a Fibre Channel HBA (we use ATTO FC-82ENs mostly but ATTO has them up to 32 gig) and an external fibre channel array (the newer Quantum/DotHill ones have 4x 16 gig fibre connections), set up your LUNs and go nuts. I have seen this used a few times but never for personal use because it gets super expensive and is a bit more complicated to set up.

So I have (raw) footage from a C300, a Sony F3, a Sony FS7 and a Red Epic, 1080, 2K, 4K and 5K. To put in bluntly: both Final Cut Pro and Adobe Premiere get angry (in OpenCL rendering) with the Red Epic 5K footage even off the SSD. This is probably due to my lower end graphics card and only having 16 gigs of RAM, both on my upgrade list. Also I'm playing with camera raw and not transcoded media, I'm sure if it was 5K ProRes it wouldn't get annoyed. The 1080, 2K and 4K played back without any real issue in either program. In Premiere I could change the playback resolution so I was able to get the 5K footage more under control by playing it back at 1/4th and it still looked fine. I didn't find this option in FCP X. I personally find Premiere to be better from a UI and options perspective. Also I didn't bother rendering in Premiere because that would have taken a serious amount of time to render camera raw and who renders footage these days!

I couldn't test Metal playback in CC 2017 because it's a bug introduced in 10.12.2! https://helpx.adobe.com/premiere-pro/kb/known-issues-premiere-pro-cc-2017.html
 
Last edited:
Joined
Dec 26, 2016
Messages
123
Motherboard
Gigabyte GA-Z170X-Gaming 3
CPU
i7 6700K
Graphics
RX 480
Mac
MacBook, MacBook Pro
Classic Mac
Power Mac
I found an very interesting load on information based on Windows machines but the topic "Premiere storage concepts" here: http://ppbm7.com/index.php/tweakers-page/84-disk-setup/95-disk-setup

I agree to your advise not to use a striped raid on the boot volume. It does not help editing performance, only doubles the risk of catastrophic data loss.
To be honest, i have to break a "dont's rule" for this video editing machine by installing non-video related software on an editing machine, like an Office suite and probably a few games since this will be my only one desktop.

Yes i planned with an external NAS, probably also two some months later when i finished this project.
One NAS as the primary system and film+photo file backup solution based on Gigabit RJ-45. For now i don't know if RAID 5 or 10 and i haven't thought about to overall netto capacity. Just a hint, i saw that the logic board has an external USB 3.1 port if i'm right. But i think Gigabit Ethernet is just ok for this usage.
Maybe a 2nd small NAS just as a data grave directly attached via USB* to the system (i'll come later back again on this).

My gear is a Sony FS700RH and a Sony A7S. An external recorder like an Atomos Inferno or a CD 7Q+ for UHD/4K DCI will come.
Codecs like AVCHD, XAVC, ProRes and maybe some GoPro stuff will be used which are all disk & CPU/memory intensive according to the codec table shown in this link.
For now it's just 1080P and UHD. But 4k DCI will come when i buy an external recorder.

Based on the linked article (Practical disk setup and Guidelines of disk usage) i thought on this internal storage concepts:
(i think it can be somehow adapted for all kinds of NLEs like Premiere/AE, FCPX and Lightroom somehow with it's database)

All based on internal SATA 6G drives

C1. 3 disks +1
SSD OS+Applications -> 250 or 500GB
HDD Media, Projects -> 2TB
SSD Previews, Media cache, Exports -> 500GB
HDD Data grave for daily stuff like downloads and all other junk 2 TB to cancel the 2nd NAS

C2. 4 disks +1
SSD OS+Applications -> 250 or 500GB
HDD Media, Projects -> 2TB
SSD Media cache -> 500GB
SSD Previews, Exports -> 250GB
HDD Data grave for daily stuff like downloads and all other junk 2 TB to cancel the 2nd NAS

I thought on C1 and C2 to choose a 250GB SSD for Media and Projects. But i fear this wouldn't be enough space when i start to use external recorder footage. So i don't know if this HDD according to the information below could be a bottleneck?

You're welcome to correct me if the HDD/SSD combos and the size doesn't make sense.


I've chosen this depending on this...

The kind of files used during editing are, in order of their need for speed:

  1. Media cache & Media cache database files, created on importing media into a project. They contain indexed, conformed audio and peak files for waveform display.
    Typically small files, but lots of them, so in the end they still occupy lots of disk space.
  2. Preview (rendered) files, created when the time-line is rendered for preview purposes, the red bar turned to green. Read all the time when previewing the time-line.
  3. Project files, including project auto-save files, that are constantly being read and saved as auto-save files and written when saving your edits.
  4. Media files, the original video material ingested from tape or card based cameras. Typically long files, only used for reading, since PR is a non-destructive editor.
  5. Export files, created when the time-line is exported to its final delivery format. These files are typically only written once and often vary in size from several hundred KB to tens of GB.
When you are doubting which category of files to put on which kind of disk, especially when using both SSD's and HDD's, keep in mind that the speed advantage of SSD's over HDD's is most noteworthy with the Media cache & Media cache database. These files are frequently accessed, are small and there are many, so reducing latency and seek times and increasing transfer rates pays off by putting these on a SSD, rather than on a HDD, even if it is a raid0. Export files can go to the slowest volume on your system, since you only export once. To help you decide, I have added priority rank-numbers for speed, with 1 for the fastest volume and 5 for the least speed-demanding category.

What happens when you have less than 6 disks / volumes? You combine certain files types on the same volume, for instance project files and export files on the same volume. That entails a higher fill rate on that volume and thus more 'fill rate' degradation, lowering the sustained transfer rate on that volume. In addition the overhead from the OS increases, because the Windows house-keeping tasks, like updating file allocation tables and access and modification time-stamps for each file used, increases due to the extra number of modified files. Of course the main bottleneck still is the half-duplex problem, waiting for reading to finish before writing can occur and vice versa.

Why did I use project and export files to be combined in the example above? Because the project files are relatively small, as are the export files, so when you combine them on a single volume the performance degradation from 'fill-rate' is relatively small, especially if you consider that on writing the export files to the volume, the project file is no longer accessed. The work has already been done. An alternative approach might be to combine Media Cache, Previews and Exports on the same volume, because Previews are not used during Export nor is Media Cache, so Exports are the only (write) activity on this volume. These kind of considerations ripple through to the table below, which shows how one could allocate the various type of files to the available disks or volumes.
 
Last edited:
Joined
Dec 26, 2016
Messages
123
Motherboard
Gigabyte GA-Z170X-Gaming 3
CPU
i7 6700K
Graphics
RX 480
Mac
MacBook, MacBook Pro
Classic Mac
Power Mac
I tought on one of the following combinations…..

3 disks + 1:
250GB SSD OS+Applications Samsung 850 EVO
2TB HDD Media+Projects WD Black
500GB SSD Previews, Media cache, Exports Samsung 850 EVO
2TB HDD etc WD Blue
Costs = 482,35€


4 disks +1:
250GB SSD OS+Applications Samsung 850 EVO
500GB SSD Media, Projects Samsung 850 EVO
500GB SSD Media cache Samsung 850 EVO
250GB SSD Previews, Exports Samsung 850 EVO
2TB HDD etc WD Blue
Costs = 590,84€

250GB SSD OS+Applications Samsung 850 EVO
250GB SSD Media, Projects Samsung 850 EVO
500GB SSD Media cache Samsung 850 EVO
250GB SSD Previews, Exports Samsung 850 EVO
2TB HDD etc WD Blue
Costs = 526,40€

250GB SSD OS+Applications Samsung 850 EVO
2TB HDD Media, Projects WD Black
500GB SSD Media cache Samsung 850 EVO
250GB SSD Previews, Exports Samsung 850 EVO
2TB HDD etc WD Blue
Costs = 577,24€
 
Joined
Dec 26, 2016
Messages
123
Motherboard
Gigabyte GA-Z170X-Gaming 3
CPU
i7 6700K
Graphics
RX 480
Mac
MacBook, MacBook Pro
Classic Mac
Power Mac
Ok i ordered all components except the Display today and went with this internal storage config:

250GB SSD OS+Applications Samsung 850 EVO
2TB HDD Media, Projects WD Black
500GB SSD Media cache Samsung 850 EVO
250GB SSD Previews, Exports Samsung 850 EVO
2TB HDD etc WD Blue
 
Joined
Jun 22, 2013
Messages
66
Motherboard
GA-Z77X-UP5 TH
CPU
i7-3770K
Graphics
HD 7970
Could I ask, what if Premier were not a part of the equation and FCPX was your only NLE? Would your drive configuration change?

Seems like FCPX has done away with the need for separate drives as a scratch disk, etc. but I'm sure I don't know much when it comes to drive optimization.
 
Top