Contribute
Register

RAID O on M2.NVME SSD

Status
Not open for further replies.
Joined
May 8, 2016
Messages
85
Motherboard
Gigabyte Z170X-UD5
CPU
i7-6700
Graphics
RX 580
Mac
  1. MacBook Pro
  2. Mac mini
Just a question. My machine currently has a Samsung M2.NVME SSD (the 960 EVO) which works well as the Main system (boot and all data) drive on Catalina. The machine has two M2 ports... and was giving some thought to adding another one in RAID 0 and attempting to use that to improve speed. There seems to be lot of different commentary on RAID, some of it old,.. and given both Catalina and also the Mac Pro I wondered what the latest thinking is. Is such a setup likely to work? (I see comments about using RAID 0 for data only.. but wanted to attempt it for the whole system). Much appreciate any thoughts. Thanks
 
@carljharris3141,

You need to bear in mind the 20 PCIe lane limitation (X16 PCIe + X4 DMI V3) on all Intel Desktop Class CPU's.

If any of the M.2 ports are serviced by the CPU (IE: Directly connected to the CPU) then installing a NVMe will switch your GPU to using x8 lanes rather than x16 which will reduce its performance by 1-5 %.

If the first M.2 port is serviced by the CPU and the second one is serviced by the PCH then the second M.2 will be sharing bandwidth with all the devices routed through the PCH (ethernet, audio, all USB ports, all SATA ports ..etc) which will severely limit it's speed compared to the M.2 port serviced by the CPU.

In that instance if you created a RAID 0 of both NVMe's, then the M.2 serviced by the CPU will be slowed down by the M.2 serviced by the PCH when the system is busy.

The only time using RAID with NVMe SSD's makes sense IMO is on X299 or Xeon systems as those CPU's have many more PCIe lanes available that are serviced by the CPU so you will get the full benefit.

Check your motherboards manual, there should be a block diagram which shows you how the PCIe lanes are distributed and which M.2 ports are directly connected to the CPU and which are routed through the PCH.

The short version is that if you want to get the real benefit of using two M.2 NVMe SSD in RAID 0 then both M.2 ports should be serviced directly by the CPU but it will come at the cost of reducing your GPU to x8 PCIe lanes.

If you have any other PCIe cards in the system then one or possibly both M.2 ports may be reduced to x2 PCIe lanes having the performance of the NVMe(s).

Cheers
Jay
 
Last edited:
@carljharris3141,

You need to bare in mind the 20 PCIe lane limitation on all Intel Desktop Class CPU's

If both M.2 ports are serviced by the CPU (IE: Directly connected to the CPU) then installing another NVMe will switch your GPU to using x8 lanes rather than x16 which will reduce its performance by 10-15 %.

If the first M.2 port is serviced by the CPU and the second one is serviced by the PCH then the second M.2 will be sharing bandwidth with all the devices routed through the PCH (ethernet, audio, all USB ports, all SATA ports ..etc) which will severely limit it's speed compared to the M.2 port serviced by the CPU.

In that instance if you created a RAID 0 of both NVMe's, then the M.2 serviced by the CPU will be slowed down by the M.2 serviced by the PCH when the system is busy.

The only time using RAID with NVMe SSD's makes sense IMO is on X299 or Xeon systems as those CPU's have many more PCIe lanes available that are serviced by the CPU so you will get the full benefit.

Check your motherboards manual, there should be a block diagram which shows you how the PCIe lanes are distributed and which M.2 ports are directly connected to the CPU and which are routed through the PCH.

The short version is that if you want to get the real benefit of using two M.2 NVMe SSD in RAID 0 the both M.2 ports should be serviced directly by the CPU but it will come at the cost of reducing your GPU to x8 PCIe lanes.

If you have any other PCIe cards in the system then one or possibly both M.2 ports will be reduced to x2 PCIe lanes having the performance of the NVMe(s).

Cheers
Jay
Thanks Jay. That's incredibly helpful detail. I checked my motherboard manual and as you say one M2 is serviced by the CPU and one by the PCIe lanes (in fact it knocks out slot PCIEX4). My manual discusses this on page 29/30 (copy attached for those interested). But in short reading what you've written if one has a GPU (as I do) RAID-ing in O two M2 probably isn't worth it..
 

Attachments

  • mb_manual_ga-z170x-ud5_e copy.pdf
    26 MB · Views: 74
I checked my motherboard manual and as you say one M2 is serviced by the CPU and one by the PCIe lanes (in fact it knocks out slot PCIEX4).


@carljharris3141,

Unfortunately there is no block diagram in your manual so i'm not 100% sure but my best guess is as follows :-

M2D_32G = M.2 Slot Directly serviced by the CPU
M2H_32G = M.2 Slot serviced by the PCH

Aside form the letters in the port name, the fact that M2D_32G is nearest the CPU would also suggest that it is the M.2 port directly serviced by the CPU.

You might be able to confirm this on the Gigabyte forums, or even email Gigabyte themselves.
If the above is true then you will get the best performance form your existing M.2 NVMe SSD in M2D_32G.

Cheers
Jay
 
@carljharris3141,

Unfortunately there is no block diagram in your manual so i'm not 100% sure but my best guess is as follows :-

M2D_32G = M.2 Slot Directly serviced by the CPU
M2H_32G = M.2 Slot serviced by the PCH

Aside form the letters in the port name, the fact that M2D_32G is nearest the CPU would also suggest that it is the M.2 port directly serviced by the CPU.

You might be able to confirm this on the Gigabyte forums, or even email Gigabyte themselves.
If the above is true then you will get the best performance form your existing M.2 NVMe SSD in M2D_32G.

Cheers
Jay
Jay - Thanks. I think you're right on this. As it happens I'd had the NVMe SSD connected in slow M2H_32G slot for some reason. I moved it to M2D_32G. And this makes about a 10% improvement in speed (read and write) on this setup. It's doing about 1.88MB/s Write and 2.22MB/s read -- not amazing but perfectly fine.
 

Attachments

  • Screen Shot 2019-12-31 at 4.13.47 PM.png
    Screen Shot 2019-12-31 at 4.13.47 PM.png
    859.8 KB · Views: 89
As it happens I'd had the NVMe SSD connected in slow M2H_32G slot for some reason. I moved it to M2D_32G. And this makes about a 10% improvement in speed (read and write) on this setup. It's doing about 1.88MB/s Write and 2.22MB/s read.


@carljharris3141,

I have MacOS installed on a Samsung SM961/PM961 NVMe SSD which is the High-End OEM version of the Samsung Evo 960 NVMe that you have and is meant to be just a tad quicker for Reads and a bit slower for Writes according to Samsung specs.

I get 1.47MB/s Write and 2.67MB/s Read using BlackMagic Disk Speed Test with 5GB block size, so i would say that your R/W speeds are spot on.
Screenshot 2019-12-31 at 17.13.57.png
With regards to the other M.2 port on your motherboard (M2H_32G), as it is routed through the PCH it has to share the DMI V3 link to the CPU with all the other devices connected or routed through the PCH. This will include :-
  • All the SATA ports
  • Ethernet
  • WiFi/BT (if available)
  • All the USB ports
  • The on-board Audio interface
  • Any other PCIe device that is in a PCIe slot served by the PCH (usually the X1 and X4 slots).
As a result the performance of M2H_32G will vary widely depending upon the activity of all the other PCH connected devices.

Where as M2D_32G has a dedicated x4 PCIe link to the CPU that is not shared with anything. If you where looking for more R/W speed (hence your question in the OP) then installing something like the latest Samsung 970 Evo Plus would result in a decent performance boost (if you really need it).

Cheers
Jay
 
Last edited:
Where are you all getting this, that there's CPU-connected M.2 on a Z170 board?

Skylake is 16 PCI-e lanes on the CPU, and a number of lanes (20, I think) connected via the DMI - effectively, with PCI-e 3.0 speeds, 4 additional lanes.

Oftentimes, workstation workloads don't go above x8 bandwidth needed from the GPU, anyway. It can be a good upgrade to run GPU and RAID 0 SSD in x8/x8. (Though some desktop motherboards only provide 1 CPU connected slot to work with.) Since you can't take advantage of PCI-e bifurcation on this type of motherboard, it requires an expensive adapter card that has a PEX chip.
 
Skylake is 16 PCI-e lanes on the CPU.


@kylec,

Incorrect, Intel Skylake Desktop class CPU's have 20 PCIe lanes available for the PCIe bus and 4 lanes reserved for the DMI V3 PCH interconnect .. effectively 24 PCIe lanes in total, this true for all Intel Desktop Class CPU's regardless of the DMI version.

How the 20 PCI lanes are distributed/shared between the PCIe slots and M.2 slot's (if available) is dependant upon the manufacturer and specific motherboard model, you are correct in that PCIe switches are usually employed on some motherboards to allow for more complex PCIe configurations.

Since there was no block diagram in the manual provided by @carljharris3141 in post #3, I only speculated that one of the M.2 ports may be directly serviced by the CPU based on its location on the motherboard and its name, please see post #4 for the full details :-

Cheers
Jay
 
Last edited:
I'm not going to dig thru spec sheets just to prove you wrong, but there are 16 lanes plus the DMI, not 20 lanes plus the DMI.
 
@kylec,

Incorrect, Intel Skylake Desktop class CPU's have 20 PCIe lanes available for the PCIe bus and 4 lanes reserved for the DMI V3 PCH interconnect .. effectively 24 PCIe lanes in total, this true for all Intel Desktop Class CPU's regardless of the DMI version.

How the 20 PCI lanes are distributed/shared between the PCIe slots and M.2 slot's (if available) is dependant upon the manufacturer and specific motherboard model, you are correct in that PCIe switches are usually employed on some motherboards to allow for more complex PCIe configurations.

Since there was no block diagram in the manual provided by @carljharris3141 in post #3, I only speculated that one of the M.2 ports may be directly serviced by the CPU based on its location on the motherboard and its name, please see post #4 for the full details :-

Cheers
Jay


You are incorrect Jay. The 115x intel CPUs only have 16 PCI lanes out of the CPU, AM4 features 16+4. Here is my supporting evidence.

In the image you can clearly see only 16 CPU lanes and the three options for dividing those up, the 8x,4x,4x being one that Apple has used and explains why Apple has had better thunderbolt since they attach it directly to the CPU. AM4 platform features 24 lanes total from the CPU. You are describing AMD's platform Jay.
 
Status
Not open for further replies.
Back
Top