WARNING ! BEWARE: THis post might seriously damage your mental health So, after 4 days of configuration assembling (SR2 replacement by newest SRX) i'm back. :D i'm finaly able to make further tests in order to better make me aware of the totally wild issues, stupid and easy engineers choices. i won't deal with my assemblies, watercooling... here. I 'll maybe create a dedicated topic if i get the motivation to. This post is dedicated to motherboards issues and isn't a review of the board. 3 main problems: I- THE BIOS There is a real lack of options comparated to Z9PE.
All available bios options menu by menu screens. Nice job ! :D
The several available option that are borked for most of them. 2 cases : - The option, once validated, stay desactivated or you encounter a defaut bios values loading avec rebooting.
( PCIE line modification options, or others PCIE under options, Vdimm, VCPU... )
- Other options make more or less randomly totally crash the computer with numerous error codes like 19 or 6A at the first reboot.
( Vdimm, manual mod MHZ ram, advanced CPU option like power mods...)
Due to the hopelessness to overclock XEON CPUs, we no longer really need advanced CPU's option. One of the rare options it would have been interesting to have, is the possibility to manage VCPU in order to undervolt it.
By default, VCPU for 2687W CPU is 1.25V. And with 0.9V, it as stable as default frequency. ( it has been tested with mono cpu configuration ( one 2687W) on EVGA SLI with BOINC software runing during 2 days.
Yet, in the case too, EVGA suxxxxxx AGAIN! VCPU option just allow +/- offset with only +/- 160mV.
So gotta manage with useless "overvoltage" in load that give us pretty high temperatures whereas it would have get colder with lower VCPU.
VDimm: Whatever bios version, this translates in my case by : either a non appication or a 6A or B7 boot bios error.
This must forbid any RAM o/c over 1600 MHz.
Clock over 1600 with XEON is possible, despite TIN says.
I had tested several XEON CPUs on various mono CPU motherboards and it's possible to get over 2400MHz RAM frequency ( done by Christian).
But still impossible on SRX. Another very annoying thing: - We have about about 1/2 seconds to hit SUPP to get bios setup.
Contrary to SR2, where while Areca controller's checking, we had enough time to hit SUPP, on SRX, this doesn't work.
- We don't have post screen that enable us to know CPU frequency, CPU/GPU number...
Impossible to disable EVGA post logo or i haven't been able to find the option among the four available......
- No option concerning 2 mini SAS port.
- ES CPU still not managed. whereas EVGA X79 bios programmed in SRX EEPROM with mono CPU configuration => this works !
- Concerning BLK overclocking for a psychological o/c (lol) , it's totally impossible with numerous PCIE devices plugged in the motherboard.
- Voltages sensors completely suxxxx
*Same value mesured in bios, normal.
Plenty of compatibility problem, Bsod without any reason..... regularly encountered but making a list of them would be to long....
The two most serious issues made me very busy and are real pain is the ass.
II- PCIE slots So for this part you gonna have to hang on, this is getting more complicated !
it's unfortunately necessary to explain everything to clarify things and better apprehend the choices/craps of the silly SRX architecture, 'cause it's not the 11 pages EVGA manual that will help us :D
I also could have hurl that :
like that, without any explanation and details. But some of you would have understood what i meant. So, sorry for the post's length, so hang on !
In my case, i want to be able to (for some personnal reasons) to use those 7 PCIE peripheral without any concession: -Areca 1231ML (8X 1.0) -Areca 1880IX (8X 2.0) -Spliter PCIE actif (4X 1.0) -Controleur IB (8X 2.0) - ATI 6990 (16X 2.0) -ATI 6990 (16X 2.0) -nvidia (470GTX) To make it simple, Xeon owns 3 PCIE Slots (16x,16x,8x) managed by CPU's internal controler (IOH).
Those lines being physicaly splitted depending on the need, in order to get more physical slots ( with less PCIE line/slot, or to link the PCH, LAN chip, sound, SATA/SAS....)
Settings the screen shot thereafter are directly the IOH's available modifications .
In the Device Manager, can be found out,
for each CPU, the existing physical PCIE slots with
maximal preallocated PCIE lines per Physical PCIE slot.
Exemple 1: You can notice that four lines have been setup to link CPU to PCH.
Exemple 2: In his case, if the slot has 4 dedicated physical lines max. So the GTX470 is managed with 4x PCIE lines instead of 8x.
Logical but suxxxx
EVGA precices tha PCIE number 6 is a 4x physical one.
Now let's complicate things. EVGA decided to use only
one PLX chip .... the PEX8747 .
PDF: http://www.google.fr/url?sa=t&rct=j&q=plx%20pex8747%20.pdf&source=web&cd=2&ved=0CFEQFjAB&url=http%3A%2F%2Fwww.plxtech.com%2Fdownload%2Ffile%2F1824&ei=vyHmT_eXGYK4hAfyt4XcCQ&usg=AFQjCNFOHxVdFGndo-UHaowiclyhaegsNg&cad=rja This chip is simply excellent 'cause it has amazing functions, to be dicovered in the PDF.
*We finally have a public chip that allows active PCIE mux + other pretty nice options !
Let's focus on mux part.
How and with how many lines SRX has been wired: *PCIE slots presentation under HWinfo 64 doesn't correspond to slots order (1 to 7) on the motherboard.
As we noticed in the 1st example, 4 lines PCIE slot was used to make connexion between PCH (However, that doesn't represent a physical slot on the motherboard).
PORT 1 (4x max): corresponds to the 1rst physical port directly linked to CPU's IOH :
it currently manages GTX470 graphic card (i willingly plugged it on this slot for miscellaneous reasons...)
PORT 2 (8x max):
On this slot is plugged an ARECA 1231ML rev2 thus 8x 1.0 hence the "current link speed: 2.5Gb/s"
Here too, nothing abnormal. PCIE still managed from IOH.
PORT 3 (8x max):
This slot is used by my active splitter PCIE 1x4X to 4x 1X
Once again, nothing abnormal. PCIE still managed from IOH.
PORT 4 (16x max):
We finally focus on 16x CPU slot that is in this case used by the famous PLX chip that allows to get 4 physical PCIE slot ( so 8X/slot, see motherboard layout in the PDF), from one PCIE 16x from CPU.
Consequently, the four last PCIE slot of the motherboard (4,5,6,7) all stem from PLX chip.
That give us real pain in this ass is the case of graphic card or other device is plugged on it:
mainly because of: generated latencies, Bandwidth bottleneck, PCIE peripheral allocation by both CPUs, PCIE lines limitation.....
Because of that, numerous issues appear: Areca 1880IX is really mismanaged concerning PCIE lines. When it's plugged in 8X PCIE (managed by IOH), it appears to be detected as 2X PCIE !! ( yeah so crappy...)
whether 4X in best case, whatever the used PCIE slot ! On one of the PLX managed PCIE, it stays @4x too(2.0 too) we gonna have to settle for that.
I've never been able to get 8X speed.
Damn it ! EVGA, we really need a fix for that !!!! So 6990 are managed in 8X 2.0 wether 16X 1.0 !!!
So pitiful for 2 GPUs Graphic Cards.
For IB controler (contrary to SR2, this time it uses 2.0 bandwidth) :) citation: So real PCIE distribution ( and this time, physical placement order on the motherboard, and from north to south) gives us:
According to EVGA:
Slot1 : 16X/8X Slot2 : 8X Slot3 : 16X/8X Slot4: 8X Slot5: 16X/8X Slot6: 4X Slot7: 8X
And according to me:
Slot1 : (direct IOH CPU1) 8X Slot2 : (direct IOH CPU1) 8X Slot3 : (direct PLX) 16X* /8X Slot4: (direct PLX) 8X Slot5: (direct PLX) 16X*/8X Slot6: (direct IOH CPU1) 4X Slot7: (direct PLX) 8X *: Only if 2 slots on 4 managed by PLX chip are used. You can notice that: checking order @boot seems to be inverted like on SR2 ( from 7 to 1), due to the fact that PCIE slots managed by PLX chip are not all juxtaposed. This makes this order complicated.
And this time, no bios option to invert PCIE checking order.
You can also notice that: Lines allocation bios options, are almost useless 'cause all PCIE are 8X limited. The only exception is when two 16X PCIE devices are plugged in 2 PCIE of the 4 managed by PLX. They will run @ full speed (16X 2.0). (to be confirmed, haven't tested myself).
But PLX chip will automatically define speed PCIE slot speed depending on the plugged device. So BIOS lines options are useless too).
Hmmmm wait wait wait, we only have PCIE lines from one CPU as you can notice ! Yep ! That is the problem ! 2nd CPU's lines aren't used: such a shame ! 
Let's notice that half of ther PCIE lines options are completely useless.
The question is:
Why EVGA haven't followed the others like Asus and haven't used ALL PCIE lines from two CPUs in order to manage 6 or 7 PCIE ???
They could have use 2 chips (1 by CPU).
Or simply dedicate direct IOH PCIE slots to CPU 2 that would have been able to manage them with 16X and 8X speed.
Asus and Supermicro are clearer concerning this point: 
Using PLX chip is a pretty nice idea for mono CPU, allowing to get more PCIE(7) and intercom functions Graphic Cards VIA PLX chip....
But on 2 CPU motherboards, using all these PCIE lines.... so crappy.
Why these technical choices ???
Easiest way to get shorter development time or simply reducing costs ?
I would be pleased to get EVGA's explanations about that it would be welcome!
To conclude, this motherboard looks like EVGA X79 Classified where another socket would have been added to be able to manage one more CPU....
That would explain BIOS compatibility between this motherboard and the SRX.
III - RAM Management Good fortune or not, 16GB sticks are fully functionals on this motherboard.
And 196GB are managed.
@1600 MHz it doesn't work correctly.
BUT Bandwitdth really sucks !!! I think there is something wrong with NUMA management and others 'cause here's the results : For your information, on Z8PE D18 board (with 147GB tri-chanel DDR3@1066), 100Gb ramdisk took 5 seconds to be created.
On SR2, it took more time even for only 10Gb ramdisk.
But on SRX it takes a little more time :D
Does CPU2 address is own RAM ?
Like absolutely NO memory management options ( mirroring, single, sparing, interleave...) are available in BIOS, i have a real lack of informations concerning the origin of the problem.
150GB ramdisk creation attempt. Begining... 
10 minutes later, only about 20GB have been addressed. 
That gives us about 100-150 MB/s. Sooooooooo performing for RAM ! :D
Usefull information: 
* I insist about this point: There is no fully reliable RAM bench software.
Most of time very dependent of CPU o/c, CPU cache memory, bad NUMA management, I/O tests variability....
So it canot be directly workable, and i put AIDA64 cache test just for your information.
ECC seems to be not functional too, and no bios option to activate it..... Test @1600MHz OK. But manay BSOD/auto reboot under windows especially when bandwidth hungry applications are running .
Must be due to a little lack of VDimm.
Another issue due to that: During max compression, 180Gb RAM are required. except that computations don't wait that dictionnaries be completely loaded into memory to start.
Result: Dictionaries transfer to cache memory is so long, that calculation is done before caching ends.
Maybe this bad memory management comes from Windows. Yeah yeah, drivers, softs.... are up to date. I prefer mention it before any genious guy makes such magical proposal.
Still testing this but without any RAM option, quickly out of solution.
Something weard, RAM @1600MHz crash under windows during compression or other taks.
=> not stable, surely 'cause VDimm not manageable, Vdrop.....
By using multimeter, i measure 1.35V VDimm so that Bios says 1.4V.
Same for 12V : BIOS 10.7V, and multimeter : 11.87V.
By getting this SRX, i hoped better RAM management , and faster PCIE thanks to Direct CPU to peripheral link.
I'm sooooooooo disappointed.
Add to this, the impossibility to o/c XEON, the change doesn't worth it in my case !
Without any reason, after bios setup modification, randomly, i gonna loose PCIE periph, most of time both 6990 !
Complete shutdown after bios setup changes fix the issue. But that's not really practical.
Several good points to avoid negative ending : -S3 mode functional even if too few annex options. ( while i'm pissed off ARECA still doesn't manage it) i'm working on it with ARECA to try to find out a fix.
- Jumper replacement (SR2), by dipswitch concerning PCIE/Ram slots (de)activation.
- Ram slot dipswitch even if ther are 8 instead of 12.
- Still 3 BIOS with switch
To put it in a nutshell, For all those major bugs and others bull is what I say!
Jacob make a REAL BIOS EVGA X79 BIOS flashed in SRX EEPROM, allows many other options comparated to SRX BIOS, What the heck !!!!???
Don't wanna wait 7 months another time for these majors get fixed.
-Find a solution to get good Areca 1880/82 management.
- Give us explanations about this PCIE architecture.
And more ...
Thanks for reading :)
post edited by mosie - 2012/07/14 16:15:43