kazu4009
X299DARK 10980XE finally let me just use my XMP profile on 4x8GB G.Skill F4-3800C14Q-32GTZN
 
XMP 3800 CL14-16-16-36-666 1T
https://i.imgur.com/OQEbHCE.jpg
 
@4000 CL17-18-18-28-320 1T
https://i.imgur.com/qkSFTqv.jpg

With all due respect what that has to do with the topic?

EVGATech_JoseC
For clarification purposes, we are looking into performance difficulties that ZoranC is experiencing. Our motherboard team is looking into this and we hope to get an answer soon. 
 
-Jose C. 

Hi Jose, please notice it is not just me and not just with 32GB DIMMs. Please see following post by xuqi99 that is experiencing same issue with various DIMM kits:
 
https://forums.evga.com/FindPost/3042046

Thank you for the info. 

EVGATech_LeeM
Alright guys, bottom line is that it's when you're using 4 channels.  Period.  It doesn't matter the brand, nor does it matter the processor brand.
 
We also asked G.Skill to look into it and they saw similar results in Performance Test to what's been reported here when they tested with a competitor's X299 motherboard and also on a TRX40 motherboard.
 
It's important to note that it's not the overall memory performance that drops when you're using 4 channels; it's just the Memory Read Uncached performance.  Otherwise, there's a notable memory performance increase overall when using the full quad channel.
 
Here is what G.Skill provided with a TRX40 board using 8GB DIMMs at 3600Mhz (2 DIMMs, 3 DIMMS, 4 DIMMS)
Overall, this could imply that there are issues with multiple motherboard platforms, or that the testing utility may not be testing 4 channel platforms properly.  Either way, at least you all have some new data to consider.

so what exactly im looking at there??

heres my test

btw may i ask why the boards/bios dont have a locked cache multi option ofr lowest and max option? my asrock one have this option...  This board goes up and down like it aint no business in the cache..

One issue here is that I'm not nearly as familiar with this benchmark, as I've been with others in the past.  I can't say, for example, that the benchmark may be unduly influenced by factors such as CPU speed, memory latency, memory speed, memory available, or other tweaks that people often set or do to improve benchmark performance.  Assuming that the benchmark is fairly balanced, then we still have to account for both the programming and OS variables, before we get to actual performance values.
 
I preface my post by saying that because although we know that at least one performance metric used by this benchmark is peculiarly affected by adding a 4th DIMM into a quad-channel memory configuration - repeatable among different chipsets and platforms - we may not necessarily know if another factor further influences performance, such as the amount of memory used (e.g. 128GB) or other factors.  Given that we and an independent memory manufacturer have produced similar results, it suggests, as I noted, that there may be either questions about how the benchmark determines performance or that the memory controllers on quad-channel supported Intel and AMD processors appear to suffer a slight performance bottleneck in a specific metric.  For the first, this is a benchmark tool provided by a third-party manufacturer; for the second, that would be far beyond our capabilities to correct.
 
With regards to your post, and edited for responses:
 

ZoranC
 
@EVGATech_LeeM Thank you for the update. I have following comments and questions, please:
 
As I am looking at the data you shared I do not feel that particular data can result in 100% reliable conclusion -SAME- issue happens with other brands and platforms too, nor in statement “… not the overall memory performance that drops when you're using 4 channels; it's just the Memory Read Uncached performance.  Otherwise, there's a notable memory performance increase overall when using the full quad channel”.

Let's be clear, this isn't just our testing, it's also testing by G.Skill.  Although I've provided an abbreviated answer above, that does not imply that limited testing was done.  G.Skill looked at the claims being made in the thread here and noted that this performance is in line with what should be expected.  That does not mean that some people have a slight variance over their overall memory performance, but it does mean that this "low performance" is, in fact, standard performance.
 

ZoranC
Reasons why I feel this way can be explained by looking at the figures[...]
 
In the end I do NOT see “notable memory performance increase overall when using the full quad channel”. To be specific I do see it for TRX40 but not for X299 Dark, at least not when it comes to any operations that involve reads that do not fit in the cache. 5% gain on database operations and 13% gain on writes can’t be, IMHO, called notable. Overall TRX40 seemed to have much bigger gains and much smaller losses than X299 Dark when going from dual to quad memory channel.

I think you're getting too far afield of the discussion here.  You're comparing anecdotal evidence and trying to fit the numbers into a theory.  The investigation into the memory performance and the conclusions of the investigation is less about the specific numbers a person should get with certain memory speed/latency, and more about whether there is a notable and repeatable memory performance dip when adding 4 DIMMs.  Comparing two different platforms to compare the performance numbers isn't the goal here; the goal is to see whether there's a performance dip on each motherboard platform respectively compared to its own numbers when using a quad-channel configuration.  
 

ZoranC 
This is where I have to wonder is posting screenshots from TRX40 helping or muddying up the issue at hand. I know very little about memory architectures but it is my understanding TRX40 mbs might employ NUMA while X299 does not, which in turn might result in better memory performance from TRX40.
 
If that is correct then TRX40 can’t be used to make any relevant conclusions.

Again, that's irrelevant.  The question is whether there is a drop in memory performance in a specific metric (or other metrics) when a 4th DIMM is added.  Per G.Skill's testing, yes, there is.  As such, it certainly shows that this is not an issue related only to the X299 platform, or that it is an issue related only to the X299 DARK. 
 

ZoranC
This leaves me with an impression that:
 
[...] 
c)  We/you should try not to introduce new variables, like NUMA, into an equation. I feel it would be much better if we were given apples to apples results, like X299 Dark with Cascade Lake-X vs. same one with Skylake-X, and X299 Dark vs. Asus or whatever.

As I noted in my earlier post, G.Skill also tested on an MSI X299 motherboard and saw a similar drop in performance for this metric.  The addition of the TRX40 board does not introduce new variables.  If anything it supports a conclusion that the "low performance" noted in this thread is not limited to the X299 DARK, or even the X299 platform, but rather any motherboards/processors that utilize a quad-channel memory configuration.
 

ZoranC 
Also, I’ve tried searching whether others too have experienced PassMark score drop when going from dual to quad channel and I’ve come across this thread:
 
https://www.techpowerup.com/forums/threads/workstation-ddr4-memory-benchmarks-ecc-vs-non-ecc-16-gb-vs-32-gb-single-vs-dual-vs-quad-channel-overclocked-vs-default-timings.257565/
 
That thread indicates that yes, others too have experienced that drop, with different Intel chipset and mb vendor too, -BUT NOT IN SAME EXTENT-. That thread’s poster had 9% drop, which is more in line with TRX40’s 5%, not anywhere near 41% that X299 Dark shows.

As much as that thread also shows that a quad-channel configuration experiences drops in uncached read times, I would caution against using benchmark data from a different chipset, with a different software version, and OS differences.  It suggests a trend, but it's hard to tell how reliable that data is to put up against our current systems.
 

ZoranC
So if your effort in the end confirms uncached read drop is to be expected my next question will be: How MUCH of a drop is to be expected as normal, is 41% on X299 Dark normal or it is excessive?

This is a much better question.  Unfortunately, I don't have an answer, and I'm not sure if there is one to find.  As I mentioned at the very top of this post, there could be a lot of factors influencing your scores.  Overclocks, lack of overclocks, etc. could also be in play.  If you have your system overclocked, I'd suggest trying with your CPU at stock, with your memory set to XMP and see if you still have the same disparity in test results.
 
If I have some time this weekend, I may be able to do some passmark runs and post the benchmarks from my X299 DARK.  At a certain point, this may be anecdotal and could be down to a lot of different factors.  Again, I'll try to look at my benchmarks and see if there's a pattern or if our numbers are way off.  
 

ZoranC 
Last, but not least, in my experience there are “lies, damn lies, statistics, and benchmarks” so I do not implicitly trust any benchmarking software. I have now come across statements that PassMark’s read un-cached favors single cores with higher clocks (why they would do that beats me).
 
Have you tried reaching out to PassMark? Better yet, do you know of a better memory benchmark that would either confirm what PassMark is indicating or question it?

There are a number of benchmarks that prefer single cores with higher clocks.  Generally, GPU benchmarks tend to do better with pure speed (excepting the CPU tests, if included), and other CPU/Memory benchmarks often do better with single cores and higher clocks (PiFast, for example).  I think a lot of it has to do with how Windows processes tasks, and that I can't imagine that using multiple cores would reduce memory latency or improve memory processing.  The less busy the CPU is, the faster the memory can handle reads/writes.  But that would be my guess.
 
No, we did not reach out to PassMark.  For benchmarks, I can't vouch for these specifically, but HWBot recognizes these memory benchmarks:  https://hwbot.org/benchmarks/memory
 

ZoranC 
In the end, what is next I can expect from you and when can I expect it please?

I'm not sure what else we can offer you at this time.  From our end, it seems that this is less of an "issue" and more "working as intended", to a degree.  Now, if for some reason you have unusually low memory performance compared to other X299 boards, then additional troubleshooting may be needed.  That's a separate issue than what this thread originally started as, which was to confirm whether or not a performance drop occurs at a certain point when 4 DIMMS were installed.  We did, in fact confirm that.  Now, moving to whether the performance drop you're seeing is average or too low is a separate question.  Let's see what my benchmarks show and we'll go from there.
 

ZoranC 
Speaking of which, could you please provide me an update on that thread about throttling I shouldn’t be having yet I do? Last update was that you are looking at it but that was a month ago and no updates since then.

Yes.  Sorry about that.  I do have an update for you, and I'll post it later today.  

zGunBLADEz

ZoranC

zGunBLADEz
so what exactly im looking at there??

heres my test

Thank you! You are supposed to compare quad channel results with two channel ones (taking two memory sticks out).
 
Which mb, CPU and memory kit is that, please?

This is a micro2 , gskill 2 sets of 16gb 3600 b dies.

What im looking at your previous settings just to let you know as i have a 32gb kit as well 3600 b dies they dont go past 4000 stable as a matter of fact i have the highest stable kit of 16gb x stick on overclock.net under 8700k an itx asus 370.

16gb kits dont overclock as good as 8gb kits so that goes to 32gb sticks as well. Also your timings are so loose trfc at 800+? All those timings aint giving you performance at all. Even at 4 chan not counting you have 3200...

Thank you! Could you please run test of just two sticks? Also, which CPU you have in?

So is my 3200Mhz Gskill Samsung bdie kit affected with a 9800x in quad channel...? What is the TLDR? There is a lot to read through and seems to make me a bit dizzy

EVGATech_LeeM
Now, moving to whether the performance drop you're seeing is average or too low is a separate question.  Let's see what my benchmarks show and we'll go from there.

 
In the absence of any further news/data from EVGA ? EVGATech_LeeM I’ve tried to see what I can figure out by myself. Obviously I am not in position to find out are my “losses” in line with what other X299 setups would have or not (for that I would have to get and test them, which I don’t have money and time for) but I could at least try to figure out is PassMark Read Uncached misleading or not.
 
That is what I have been working on since the last post. Now first results are in and experience has been interesting/educational. I will try to post it as best as I can but it is going to be long(ish) writeup so I will have to break it up across more than one post and there will be breaks in between them as I try to find time to write them.

I'm not familiar with how X299 memory channels are laid out as to whether they are daisy chained or if they use T-Topology?
Is the Dark Daisy or T-top?
 
I know Daisy chained DIMM Slot board put way more strain on the IMC than T-Top does.
 
Just spouting off here based on what I know with Z370/90 and how the channels are laid out.
 
Actually thinking about it the board should be 1DPC.
 
 

zGunBLADEz
thing is you aint see too much gain with those timings brother. trfc is way too high is what 800?? @ 3200
what setting you have your cache at ?? 
if you dont tweak timings and cache you aint seeing too much difference

stick with aida64 for mem bench theres something wrong with passsmark. Theres no way  mem latency is that low specially on amd thats my score 30-32 on 4200MHz on a heavily tweaked ram on my mobo.....
aida64 lineary scale as sticks are added and is more consistent.

Passmark Memory test is not accurate and way off.

PINKTULIP
Passmark Memory test is not accurate and way off.

I'm slowly getting to that, bear with me ... :)

At this point I started feeling whatever is happening is exposed when benchmark is single threaded. Is it exposing “flaw” with benchmarks (them not being right fit for that architecture) or something else is what I want(ed) to find out but couldn’t guess (cause I am not qualified to talk about modern memory architectures).
 
I felt everything so far indicated this might be due to how benchmark is designed so I started looking for benchmarks where I would be in charge of execution parameters rather than developer picking one (or even few) “presets” for me and taking me along for a ride.
 
Findings were slim (it seems everybody loves to benchmark games and CPU but memory not so much) and almost everything was unusable to me for at least one out of following reasons:
 
1. Precompiled executables were either not working or not available (and source code, where available, would require lots of effort to get to compiled point)
2. Approaches were outdated (as in not made for scaling)
3. Setup effort  was too high
 
That left me with just one candidate. Enter Intel Memory Latency Checker …
 
{… to be continued …}
 

Intel’s Memory Latency Checker is not the program you can just run, push the button, and be done several seconds (or even minutes) later, you really want to take your time and RTFM.
 
I guess you could always run each of its 7 modules at default settings and call it a day but I figure if you are type of a person that is interested in digging deep enough to reach for IMLC then you are type of a person that will not stop at running it default values and calling it a day, otherwise you would’ve just taken what PassMark is serving you as granted.
 
IMHO by the time you are done reading its manual and exploring power it offers you will have bigger appreciation for complexity of task at hand and understanding of it.
 
It offers you not only full control over duration of tests and size of memory block used for testing but also do you want hardware prefetchers enabled/disabled, patterns, use of large pages … all the way to which CPU(s) you want which part of test pinned on or not. For me it was enough to cement the belief that something like PassMark is far from sufficient to give complete and accurate picture.
 
In the end I ran total of 30 various scenarios. It took a while to run them and compile findings but once dust has settled results were:
 
Bandwidth Matrix, Maximum Bandwidth and Peak Injection Bandwidth modules were definitely noticeably scaling up, Loaded Latency was on average scaling up, C2C latency and was practically identical when going from 2 to 4 channels.
 
Idle Latency and Latency Matrix modules have shown drop when going from 2 to 4 channels. Average drop was 5% (with 2% being lowest and 9% highest) which seemed to depend on which CPUs were engaged in test.
 
With IMLC I’ve ran out of synthetic tests and started looking at “real world” ones …
 
{… to be continued …}
 

TiN_EE
T-routing topology or Daisy chain topology does not apply to 1 DPC boards. So there are no pins in DIMM that are shared, each pin goes to each pad on CPU directly, so there is only one topology for 1 DPC, which is point to point. And that is what on all 1 DPC boards, be it Z390/Z490 or X299 DARKs or other vendors.
 
ZoranC, no benchmark will give you true memory only speed. Benchmarks only can do so much to "isolate" the performance testing scope, but it's always going to be a mix of IMC,DRAM and CPU/Mesh components. What you can do is to only collect relative abstract figures and see how that changes with different settings. And yes, reported numbers MB/s in some benchmarks are still abstract and not represent actual data bitrate in/out of DRAM. As you already figured out, cache plays big role too. When you changing number of modules populated into board, you are not just changing how many memory channels are used, but also how memory is shared inside CPU. X299 CPUs have two separate IMCs, one per each side. You can do interesting experiment - remove 2 DIMMs on one side, and test abstract performance numbers. Then remove 1 DIMM per each side, leaving 1 DIMM per side in board. Test again, you should see different numbers, even though both of the setups are 2 DIMMs , dual-channel ;). Another thing - disable some cores, and do set of tests, and you will see that far cores (from IMC) will give you slower memory "speeds" than nearby cores, because of mesh transport delays. Anyway, lots of things involved, benchmarking accurately it's not even a pandora box but a whole pandora planet ;) And multithreaded benchmarks are whole different clusterfun, because now you have many cores fighting for access to same resource (IMC). And 3rd/secondary timings in BIOS have complex relation on what topology you have in system, and how IMCs access memory in orderly fashion. However there is no much exposed controls how one can finetune what happens before data reaches IMC, it's all Intel's(AMD's/etc) secret sauce. To have somewhat control on that, you have to be deep into 0-level programming and know how to talk to CPU, IMC, mesh controller, etc. You can do bunch of PhD's on the topic (no joking here).
 
I get it, that it bugs you, that there is no way to get actual performance figures, but in last 10-15 years desktop confusers got a lot more confusing. Count it a funny coincedence when two different benchmarks showing somewhat similar numbers/results, rather than true meaning of what you thinking. 20 people saying same thing while one person disagree does not mean that all 20 are correct. ;)
 
 

Tin it's always a pleasure reading a clear(ish?) explanation from an actual engineer them self. Thanks for sharing 

Can't trust no one. What I can add is that, most of this memory access/routing/operation is not relevant to X299 Dark specific, but apply on total for a given platform. If you look on performance of one board with chipset X and CPU A, then you will see majority of all other boards on chipset X and CPU A doing same thing, given everything runs on same clocks/settings. Better boards give more settings available to tinker with, and better power/signal quality so bad stuff start happening later, esp when overclocking, but overall behavior is same, like you expect.