Info on X299 Dark overclocking with example build.
Please help with questions at end.
Suggestions welcome.
# indicates later post with more info.
 
Refer
Kingpin Dark Guide
Intel Skylake-X Data Sheet
Intel Skylake-X Spec Update
Mesh Network Theory
Mesh Network Die Analysis
 
Goal
Stable overclock for creative/scientific/deep-learning workstation.
 
Status
4.5Ghz All 18 Cores Constant
3200 DDR4 16-18-18-36 Full Speed
Mesh Ratio 30
~82C @ Prime95 56K load
Confirmed Stable Non-AVX (AVX Testing Later)
Evaluating candidate final stable configuration with case, Titan V, and 3600 16-18-18-36 memory overclock. #75
Also AMD Radeon WX Pro 7100 in tandem with Titan V. #90
Evaluation has lasted for 18 months in regular use without issues. #92
 
Build
X299 Dark On Included Bench Stand
EVGA Supernova 1200 P2 (Eco Mode ON)
7980XE (delid, 4.3Ghz bin)
EVGA CLC 280 (USB to USBHDR1/2, Block Pump Fan Cable To CHA_FAN, Radiator Fans Direct To CPU1_FAN And CPU2_FAN)
Corsair Dominator 3200 64GB CMD64GX4M4C3200C16
Samsung 960 Pro 1TB (Slot PM1)
EVGA 1080 TI Kingpin (Slot PE1; Tentative, Titan V Later)
Gigabit Ethernet To Intel i219 NIC
Thermal Grizzly Conductonaut Between IHS and CLC Waterblock #31-35
 
BIOS
1.06 (on BIOS2)
Vcore: Override 1.175
CPU Multiplier: 45x
AVX2 Negative Offset: -4 (Untested)
AVX3 Negative Offset: -6 (Untested)
Memory: XMP Profile 1 (DDR4-3200 16-18-18-36-1.35V)
VCCIO: 1.166
 
OS
Windows 10.0.16299.192
 
Driver
NVIDIA Driver 388.71
Default Windows NVM Driver
 
Software
MemTest86 7.4 (USB Bootable)
EVGA Flow Control 2.0.4.0 (Firmware Version 2.10.0.0) (see Note)
HWiNFO64 5.7-3300
Prime95 29.4,build 5 #44
Furmark 1.19.1.0
Iometer 1.1.0 (#18)
 
Config
Nvidia Global 3D Settings Power Management Mode = Prefer maximum performance
Windows Power Plan = High Performance
 
Note
Overnight Prime95 8-4096 FFT in-place 5 minutes per test pass (two times through all tests).
 
HWiNFO64 observes 14C+ delta between heavily loaded hottest/coldest cores.
 
HWiNFO64 confirms intended frequency drop when AVX enabled in Prime 95.
--
Disable AVX in Prime95 with these entries BEFORE [Worker #1] line in local.txt configuration file:
CpuSupportsAVX=0
CpuSupportsAVX2=0
CpuSupportsAVX512F=0
---
CPU clocks down to AVX2 offset, not AVX3 (AVX-512) offset when Prime95 running without AVX disabled.
 
Running cooler radiator fans on high load drops CPU temp 12C immediately.
Coolant temperature does not immediately respond.
EVGA Flow Control shows fluid temp TO radiator, not FROM radiator.
Activating fans may drop fluid temp FROM radiator to CPU, which coolant temperature sensor does not register.
 
BIOS provides better fan control than EVGA Flow Control. #30
EVGA Flow Control fan curve maximum too limiting for Skylake-X.
Wired CLC pump to CHA_FAN for pump power.
Wired CLC radiator fans to CPU_FAN1 and CPU_FAN2.
Running stock SMART profiles based on CPU temp in BIOS for radiator fans.
 
Topic
Titan V Needs Companion Video Card - if compute matters #89
AMD WX Pro 7100 Works As Companion - handles displays while Titan V computes #90
X299 Dark LED Behavior Chart - what turns on when building #91
Choosing Good Ram - frequency/latency matters, charts provided #83
Tentative Benchmarks - results close to final CPU & GPU overclock #60-61
Liquid Metal Application - enabled 45X CPU multiplier overclock thermals. #31-35
Phantom Throttling - detected when increasing Mesh Ratio to 32 from default 30 #50
WHEA CPU Internal Parity Error -  indicated overclock instability at 45X CPU multiplier #57
BIOS Default Stock Testing - observation from Furmark crash troubleshooting #59
Dark BIOS 1.05 Issues - not supporting XMP and memory overclocks #74
Mesh Ratio Overclock Troublesome - better to focus on CPU and memory overclocks #75
Titan V And Kingpin 1080 Ti Direct Comparison On Same Board Simultaneously
Dark Case Mounting Challenge - VRM heatsink gets in way #4-8
Professional Workstation Benchmarking - using SPECwpc #76
Use Samsung NVMe Driver - Standard Windows NVMe driver terrible write speed #77
SSD Firmware Matters - slight performance differences in firmware for SSD #78
Disabling Fast Startup - optional, read to find out if you want to (link courtesy EVGA tech support)
Extend/RAID M.2 - not tested (don't boot RAID, boot M.2 slot PM1, project data on RAID PCIe M.2 card if needed)
Intel Optane PCIe SSD has best 4K random read though - from this thread #26, but thread is tedious.
Mod CLC 280 To Push/Pull - extra fans drop temps; improve sound.  Alternative to custom loop. #82
 
 
Open Question
Hypothesis on fan activated CPU temp drop without coolant temp drop accurate?
What is difference between Intel i219 and i210 NIC?
How to test AVX-512? 
Why 14C+ delta between heavily loaded cores?
Can anything fill RAM faster or test mesh ratio better than Prime95? #55
 
Closed Question
Why must EVGA Flow Control run for fans to follow curve? Shouldn't curve write to firmware on click Apply? #19
Why does HWiNFO64 hang on startup without SMBus disabled? #20
Why 23C delta between heavily loaded cores?  #2-12,21,24
 
Change
2018.02.03 - Updated 4.5Ghz overclock.  Previous 4.3Ghz overclock info moved to #42.

geninfo can you post a picture of the setup?
 
Was wondering how your cooler was sitting in regards to the cpu / pump unit

geninfo

GGTV-Jon
geninfo can you post a picture of the setup?
 

 
CPU/CLC setup and thermal paste patterns: https://imgur.com/a/6SGLr

2 suggestions - make sure the return / feed line from the radiator to the pump is on the bottom (I do not know which one is which on the CLC) to insure no air being pumped through the pump / cooler and wouldn't it be better to have the entire IHS be covered by the TIM you are using?

arestavo
Over the years with all of the Intel CPUs that I've had, an 8 to 10C temperature difference on loaded cores is normal (soldered or non-soldered chips). 23C seems a bit extreme. Maybe the TIM replacement wasn't evenly applied?
 
Or perhaps your CPU cooler's cold plate is warped slightly?

Thanks.  I asked delid vendor about TIM replacement and if 23C seems abnormal.
 
How to tell if CPU cooler warped slightly?  I cannot find non-bulk Fujifilm pressure paper.

Return/feed line already at top based on infrared thermometer measurements at line connection point to radiator and top/bottom radiator itself.  Good tip!

 
Was / Is it at the top of your radiator with it on the side as in the picture or was that a typo and you meant bottom where you want it?
Perhaps just elevate the other end so any air bubbles get trapped in the end tank and not able to pass through the hoses
 

GGTV-Jon

Return/feed line already at top based on infrared thermometer measurements at line connection point to radiator and top/bottom radiator itself.  Good tip!

 
Was / Is it at the top of your radiator with it on the side as in the picture or was that a typo and you meant bottom where you want it?
Perhaps just elevate the other end so any air bubbles get trapped in the end tank and not able to pass through the hoses
 

Yes, was typo.  I meant bottom where I want it.

Does this board have bootable nvme raid 0?

NOTE: These results were during a time when 4.3Ghz overclock and configuration was in place.  See Changes in original post.
 
More results. Pushing overclock to 4.5Ghz based on https://forums.evga.com/FindPost/2746152 .
 
Note
100% fan cooling cannot contain heat with AVX above 40x/1.155VCore.
Fan curve cooling contains non-AVX heat fairly easily at any VCore tested so far (temps < 80C, VCore < 1.16).
Not even BIOS stable at 45x/1.145Vcore. Reset CMOS.
Not stable at 45x/1.555Vcore ratio. Fails Prime95.
Possibly stable at 45x/1.16Vcore ratio with AVX -5/-7.
Need overnight Prime95 run to confirm (waiting new thermal paste).
 
Run
Ratio 45, AVX -3/-5
Vcore 1.165
Prime95 Small FFT No-AVX: 38 minutes
Temps OK
STABLE
 
Ratio 45, AVX -3/-5
Vcore 1.145
Temps ?
UNSTABLE RESET CMOS
 
Ratio 45, AVX -4/-6
Vcore 1.145
Prime95 Small FTT AVX: 2 minutes
Temps RUNAWAY (Full Fan Engagement Fails)
MANUAL ABORT
 
Ratio 45, AVX -4/-6 
Vcore 1.155
Prime95 1344K No-AVX: ~10 minutes
Temps OK
BSOD
 
Ratio 45, AVX -4/-6
Vcore 1.16
Prime95 1344K No-AVX: 50 minutes
Temps 70-75C (40-41C Coolant, Fans 1800RPM)
STABLE
 
Ratio 45, AVX -5/-7
Vcore 1.16
Prime95 Small FFT No-AVX: 30 minutes
Temps 88-89C (42C Coolant, Fans 1900RPM, 23C Ambient)
STABLE
 
Question
How to test AVX-512 with Prime95? CPU clocks down to AVX2 offset, not AVX3 (AVX-512) offset when Prime95 running without AVX disabled.
 

GGTV-Jon
Were the per core temps closer now with a full IHS coverage?

Not yet reapplied.  Will try this week when thermal paste remover solutions arrive.

EVGA Flow Control to EVGA CLC 280 firmware application problem inferred as HWiNFO64 interaction issue.
 
Inference
HWiNFO64 signals CLC for sensor information.
CLC thinks Flow Control software calling it.
CLC abandons firmware settings; listens for Flow Control direction.
HWiNFO64 provides no direction.
CLC without software direction goes fan "off" (600RPM, stops following curves).
Fan "off" state preserved in firmware until Flow Control sends new direction.
 
Note
On restart and during BIOS, CLC keeps fans running on curve if Flow Control ran last.
On restart and start Flow Control, no change in current fan speed.
On restart and start HWiNFO64 after Flow Control, CLC sensors not shown in HWiNFO64.
On restart and start HWiNFO64, fans go "off"; CLC sensors shown in HWiNFO64 (as NZXT Kraken).
Only starting Flow Control after HWiNFO64 turns fans "off" turns fans on again; CLC Sensors in HWiNFO64 show zero.
HWiNFO64 shows CLC sensor values again if Flow Control stopped.
 
Original Post Notes/Questions
Must run EVGA Flow Control for fans to follow coolant curve.
Why must EVGA Flow Control run for fans to follow curve? Shouldn't curve write to firmware on click Apply?

Do NOT replace EVGA CLC 280 pre-applied stock thermal paste with Thermal Grizzly Kryonaut.
DO tighten heat sink down properly; check/tighten thumb screws after stress tests.
Thermal paste reapplication may NOT fix core-to-core temperature disparity.
 
Note
Removed existing thermal paste with Arctic Silver Arcticlean Thermal Remover.
Applied Thermal Grizzly Kryonaut to full IHS (with included spatula, even thin spread per instructions).
After additional tightening, observed temperatures about 3C higher
Before additional tightening, observed temperatures 4-5C higher.
Tightened to "hand tight", ran some stress tests, found screws could easily tighten more with same force.
Large (23C+) temperature delta between cores still observed; possibly greater than before.
 
Original Post Note/Question
HWiNFO64 observes 23C delta between heavily loaded hottest/coldest cores.
Why 23C delta between heavily loaded cores? (awaiting thermal repaste)

geninfo
Do NOT replace EVGA CLC 280 pre-applied stock thermal paste with Thermal Grizzly Kryonaut.
DO tighten heat sink down properly; check/tighten thumb screws after stress tests.
Thermal paste reapplication may NOT fix core-to-core temperature disparity.
 
Note
Removed existing thermal paste with Arctic Silver Arcticlean Thermal Remover.
Applied Thermal Grizzly Kryonaut to full IHS (with included spatula, even thin spread per instructions).
After additional tightening, observed temperatures about 3C higher
Before additional tightening, observed temperatures 4-5C higher.
Tightened to "hand tight", ran some stress tests, found screws could easily tighten more with same force.
Large (23C+) temperature delta between cores still observed; possibly greater than before.
 
Original Post Note/Question
HWiNFO64 observes 23C delta between heavily loaded hottest/coldest cores.
Why 23C delta between heavily loaded cores? (awaiting thermal repaste)

geninfo are you up for testing other Thermo Grizzly products - for Science!!
 
Thermal Grizzly Conductonaut - Thermal Conductivity of this product is 73 W/mk and thermal resistance of 0.0021 K/W.

https://www.newegg.com/Product/Product.aspx?Item=9SIA4RE6M28090&cm_re=Thermal_Grizzly-_-2S7-00A9-00013-_-Product
 
The Kryonaught you used was rated at 12.5 W/mk and thermal resistance of 0.0032 K/W.
I was going to suggest the Hydronaught as well - but it was only rated 11.8 W/mk and thermal resistance of 0.0076 K/W - not as good as the Kryonaught you already tried and did not like
 
Edit Question: I do not recall if you said your CPU came from Silicon Lottery or if you did the delid yourself, And if from SL do they have any comments on the temp delta across the cores?

3DMark Basic Edition
Fire Strike
 
GPU OC +50 Boost, +0 Memory
26,748 (Graphics 29,641, Physics 30,647, Combined 13,913)
 
GPU OC +50 Boost, +250 Memory
26,599 (Graphics 29,548, Physics 30,619, Combined 13,674)
 
GPU OC +50 Boost, +500 Memory
25,964 (Graphics 29,911, Physics 30,937, Combined 11,639)
 
GPU OC + 50 Boost = 2025 GPU Clock
GPU OC + 500 Memory = 6008 Memory Clock 
 
Question
Why total score decrease with higher memory clock?

geninfo
Flow Control Fan Curve for *Coolant*
20/25
30/25
33/40
42/70
48/100
60/100

 
Why would you let your coolant reach 48c ?
Unless your running on processor temp for your fan control.
If you are running on processor temp your fans will bounce erratically which will cause loss of processor performance minimal but still a loss.
 
 

geninfo

EVGAFAN04
Why would you let your coolant reach 48c ?
Unless your running on processor temp for your fan control.
If you are running on processor temp your fans will bounce erratically which will cause loss of processor performance minimal but still a loss.
 

 
Thanks.  You are correct.  Had abandoned coolant-based fan curve and Flow Control software.  Wired radiator fans directly to CPU1_FAN and CPU2_FAN on motherboard.  Using default BIOS SMART fan settings that follow CPU temperature, not coolant temperature.  Updated first post.

It would have been easier to use my settings on coolant temp / now your doing what I said above.
Running on processor temp your fans will bounce erratically which will cause loss of processor performance minimal but still a loss.