That about what you get ( making serial faster by going parallel )
there is a better list posted of which ports are used/disable in BIOS for which board types by EVGA  
The manual's will also show that as well as the board art work.
 
At least this make it easy to go back and forth - if you add or remove a M.2
 
 
now if you use a non- NVMe SSD M.2 ie a SATA only M.2 I don't know if the BIOS still takes all 4 lanes ?
I only tried that on a EVGA Z270
 

hpwitten
So my question is, for the M.2 Key-M slots, will it suck up PCI-e lanes or not? And, also, since each one disables two SATA ports, does that mean it bonds the two for double the bandwidth?

Kind of "yes" in that if you install an M.2 NVMe SSD, the mobo (Z370 chipset) will allocate/dedicate X4 PCI-e lanes to it (thus disabling 2x SATA ports).  So X4 PCI-e lane bandwidth will be available to the SSD.  Whether the M.2 NVMe SSD itself can utilize that bandwidth depends on the specific M.2 NVMe SSD itself, whether it is X4 (and can use all 4 lanes) or X2 (and will only use 2 lanes ... performance a little faster than SATA SSD but slower than X4 NVMe).  This is why X2 NVMes cost less, are a little faster than SATA SSD, and get rid of cables (which is why these budget NVMes are quite popular).  This is also why X4 NVMes cost more (requires higher-end on-board controller to take advantage of the X4 lanes) and are much faster than the other SSDs.  

Point of clarification ... there is no Northbridge/Southbridge under the Coffee Lake architecture ... Northbridge/Southbridge is older architecture that was replaced in 2011 by Sandy Bridge; which was replaced by Ivy Bridge; then Haswell; then Broadwell; then Sky Lake; then Kaby Lake; and now Coffee Lake microarchitecture.
 
COFEELAKE:  Today, the Intel 8th gen CPUs (like the i7-8700K) have 16x dedicated PCI-e lanes for the CPU to communicate with GPUs (using 16X lanes) & RAM (using 4X lanes).  In *addition*; the Intel Z370 host chip itself has its own 24x dedicated PCI-e lanes set aside for IO devices only.  (H370 has 20 PCI-e lanes, B360 has 12 PCI-e lanes; H310 has 6 PCI-e lanes).  So you have a total of up to 40 PCI-e lanes using the Z370 chipset.
 

hpwitten
I suppose ultimately it doesn't matter all THAT much, as even if something does occupy the PCI-e lanes, it'll only knock a GPU back to x8, and real-world data shows there's no difference, under average workloads, between x8 and x16...but now I'm confused and curious as to the answer.

 
GPU:  This is not correct.  Storage devices do NOT use any of the PCI-e lanes that are dedicated to GPUs.  GPUs have their own separate/distinct/reserved PCI-e lanes--16X of them direct with the CPU.  Storage devices (and other IO devices) communicate with the Z370 host chip which has its own dedicated 24X PCI-e lanes.  So plugging in any drive (SATA, NVMe, SSD, HDD) would utilize the 24X PCI-e lanes from the Z370 host chip and NOT the 16X PCI-e lanes reserved for GPU.
 

hpwitten
And also, I suppose, whether m.2 is superior to standard form factor SATA. That question, I haven't found data on.

 
FORM FACTOR:  M.2 form factor (e.g. chip w/no cables) is neither superior nor inferior to SATA form factor (2.5" with cables) -- it's just different.  It should not be confused with the protocols that run on it:  for SATA devices the AHCI protocol which was built for mechanical HDDs and thus is slow, and for NVMe drives the NVMe protocol which takes advantage of PCI-e lanes and speeds--more lanes equals more speed potential.
 

hpwitten
Why are we talking about SATA lanes and PCI-e lanes like they're the same thing? Did something change in Intel's implementation recently?

 
SATA and PCI-e are not the same thing.  HSIO and PCI-e are also not the same thing.  I think here-in lies the confusion.  I will try to explain more clearly in my next post.

hpwitten
Whereas in actuality, what's happening is more like this: The two silver slots  sit on those 16 GPU lanes, and literally everything else -- the four black slots, SATA drives, the NIC, USB ports -- they all communicate with the chipset over the remaining 24 lanes. So I could throw an NVMe drive in the topmost M.2 E-key slot and not impact the GPU's bandwidth at all, and ditto with the lower M.2 M-key slots -- the only difference is they leverage AHCI?

 
Close.  So think of the Z370 host chip having 30 HSIO lanes (High Speed Input/Output lanes).  And "on top" of these lanes can run other interfaces--for example USB, PCI-e, SATA, network, etc.  So if you look at figure 3-1 again, you can see HSIO lanes #1-6 (labeled in #s in orange on top) are dedicated to USB only--and only USB runs on these particular HSIO lanes.  If you look at HSIO lanes 7-30, you can see that all these lanes support PCI-e (24 of them for Z370), with some of them being only PCI-e, while others can share with other interfaces (USB, SATA, ethernet, for example). 
 
-- For example, HSIO lanes 27-30 only support PCI-e (depicted as PCI-e lanes #21-24) for PCI-e devices only. 
 
-- Another example would be HSIO lanes #23-26 which supports both PCI-e (PCI-e lanes #17-20) and/or SATA interfaces (SATA #4 and #5).  So to use these HSIO channels, you could plug in 2x SATA devices into SATA ports #4 & 5; or you could also plug in an M.2 SATA SSD which would also use one of these SATA interfaces.  You could also plug in an M.2 PCI-e **X2** NVMe (which would use PCI-e channels #19 & 20, "on top of" HSIO lanes #24 & 25)) _at the same time_ as the 2x other SATA devices (using SATA5 and SATA6, "on top of" HSIO lanes #23 & 24).  OR you could plug in an M.2 PCI-e **X4** NVMe, which would take up HSIO lanes 23-26 and thus PCI-e lanes 17-20 ... which would disable SATA ports 4 & 5 at the same time.
 
 

hpwitten
Does this mean that the RAM will take four lanes from the 16 provisioned for the GPU, leaving 12? Or am I misunderstanding this, as well?

 
Not quite.  The GPU(s) has 16x PCI-e lanes set aside for graphics only--these are not shared with anything else.  If you have one CPU, it gets all 16 lanes.  If you have 2x GPUs and SLI, for example, then each would get 8 lanes.
 
And RAM has it's own separate memory interface and channels to communicate between the CPU and RAM--I don't know the protocol.  And the memory controller on the CPU controls communications with RAM, to include "overclocking" the RAM with XMP, for example.  This is why the ability to overclock RAM (and how far) is dependent upon the CPU, and not just the motherboard.
 
Lastly, chipsets determine the # of HSIO lanes available (Z370 with 30 HSIO lanes, including 24 PCI-e lanes), and motherboard makers make decisions on how to allocate those lanes for IO.  They can choose how many M.2 slots to support (1, 2, or 3), whether each slot can support M.2 SATA only, M.2 PCIe/NVMe only, or both (for example, my mobo has 2x M.2 slots, one of which supports M.2 SATA or M.2. PCI-e/NVMe while the other one only supports PCIe/NVMe (can't plug an M.2 SATA into it).  And this is how different Z370 mobos, for example, support different #s of M.2 devices, or have different # of SATA devices, or different # of PCI-e slots (of different sizes, X16, X8, X4, or X1).  EDIT:  Also, this is where H370, B360, and H310 host chipsets support fewer HSIO lanes, and thus fewer PCI-e lanes, and thus fewer devices.
 
A very long answer to a short question.  Does this make more sense?

I think I had a major breakthrough, actually, in my own naive understanding - thanks to your patience and that Tweaktown link.
 
My new understanding is more like this:
 
The 8700k CPU itself features the sixteen interconnection lanes which power the GPU, and aren't exposed to anything else in the system.
 
Additionally, the Z370 *chipset* features an additional thirty, which connect to all other peripherals and I/O. These lanes are completely separate from the sixteen noted in Intel's ARK database, and also separate from whatever transport mechanism links to RAM.
 
The chipset's thirty HSIO lanes are broken down as follows:
 
The first six must be used for USB3.
 
The following 24 are the "PCI Express" lanes that aren't the sixteen directly linking the CPU to the GPU (I think those sixteen used to be called Quick-Path Interconnect?)
 
These 24 lanes, though they rest on top of PCI-E, can have other services put on top of them -- I'm immediately put in mind of the OSI networking model, where PCI-E might be thought of as the Transport layer, with Session/Presentation/Application still "on top," representing multiplexing into further USB lanes, SATA lanes, Ethernet, etc.
 
Some of the Z370 FTW's PCI-E lanes are electrically switched between SATA ports 0/1 and 4/5 and the two m.2 Key-M slots in a mutually exclusive fashion.
 
(The manual says each slot is x4, but each SATA port occupies one HSIO lane, so I'm not entirely sure what's going on there.)
 
But since the SATA / AHCI protocols "ride on top" of the underlying PCI-e lane, I can stuff an NVMe SSD there and take advantage of the full bandwidth thus exposed.
 
I think I've gotten that all right?
 
It really isn't set out that clearly, though, which is kinda BS. But thanks a lot for your explanations!

hpwitten,
 
Here's a great video I just found that explains this much better than I did: 
 
https://www.youtube.com/watch?v=JVihgHoy8Sg

int3rl0ck
I have a 960 evo nvme and there is 2 problems with the installation on z370 evga (i have the K) , if you use the top m2 slot, its quite tight fit with the top video card bringing heat to both together, hotter card and the nvme.. and if you have an heatsink like i do(EK air) for the drive, it just doesnt fit. And another thing, the screw that comes with it is a gimmic joke, its Really to short, I had to remove the kind of nut screwed in the mb, and screw the nvme drive with a screw 1mm X3mm scavenged from a laptop and shim the drive with a piece of electrical tape since there was nothing to set at the proper angle without the original mobo nut.
<snip>

I also haVe a Z370 Classified k and use a 970 EVO NVMe 500GB in the top M2 slot sandwiched in nickel EK-M.2 NVMe heatsink (lowered temperatures about 10⁰C) - no problem with the fit and i'm running GTX 1080 Ti's in SLI, As for the retaining screw... I'll admit it's comically small, easy to lose and difficult to work with but it is the correct size.