EVGA

Hot!SR-2 - 96GB Secret Sauce

Page: 1234 > Showing page 1 of 4
Author
gordan79
SSC Member
  • Total Posts : 532
  • Reward points : 0
  • Joined: 2013/01/27 00:17:36
  • Status: offline
  • Ribbons : 3
2013/12/30 11:06:47 (permalink)
I've been doing a bit more experimentation with this so I thought I'd share my findings.
 
1) Single-ranked RAM won't work for 6x8GB.
2) Dual-ranked RAM can work for 6x8GB.
 
The RAM I'm using is this:
http://www.crucial.com/uk...module=CT8G3ERSLD4160B
 
Using suitable RAM is NOT sufficient to get this working. The CPU also matters. I have a pair of X5650s SLBV3 stepping. They are otherwise different - different place of manufacture and very different serial numbers.
 
The newer one was made in Costa Rica, and on that one I can POST with 6x8GB with no special BIOS tweaking (everything set to auto, 166 FSB, RAM set to 1333 (1660MHz actual), 1600 strap). It usually cycles through b8,b9 once, and on the secont attempt it boots.
The older one was made in Malaysia and that one won't POST with 6x8GB regardless of the BIOS settings.
 
Note: "Auto" voltage settings when OC-ing seem to default to:
VCore: 1.30V
VTT: 1.35V
VDIMM: 1.65V (rather high for 1.35V DIMMs, especially when they aren't being clocked out of spec)
 
Further things to test are whether the 6x8GB capability follow the CPU or the socket, but my older, Malaysian X5650 has always been a little proner to glitches (e.g. not detecting all the RAM at > 1333MHz.
 
This also seems to tally up with other people's findings - higher spec Xeons that were released later (e.g. X5690) seem to fare better when it comes to initializing all the RAM in time. It looks like even though the stepping hasn't changed, the newer silicon nevertheless includes some changes that help.
 
This is fairly clearly a BIOS bug (since Westmere Xeons are specced for 192GB of RAM each, IIRC) - it looks like the timeout for the CPU initializing all the RAM is too low. There is a reasonable chance that upping that timeout might actually allow 96GB configurations to work on this motherboard without problems.

Supermicro X8DTH-6, 2x X5690
Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
3x GTX 1080Ti
Triple-Seat Virtualized With VGA Passthrough (KVM)
#1

90 Replies Related Threads

    mpogr
    New Member
    • Total Posts : 34
    • Reward points : 0
    • Joined: 2007/01/26 01:55:44
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2013/12/31 02:54:59 (permalink)
    Mate, how would you classify the phenomenon I'm experiencing, which is being able to ALWAYS POST with full 96GB after CMOS reset, but then unable to POST after saving ANY, even default, settings into the BIOS? I haven't tried playing with my CPUs, which are both X5650s ES off eBay. That's still puzzling me, I'm just out of energy to spent more time on trying to tweak this weird mobo...
    #2
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2013/12/31 03:59:44 (permalink)
    Yes, I remember you mentioning that before. If my theory about the RAM initialization timeout is right, the chances are that what is happening is something like this:
    • When you reset the CMOS, the POST-ing up to RAM initialization takes a little longer because there is more probing and initialization going on (e.g. CPU detection).
    • This extra time gives the CPUs a few hundred milliseconds longer to initialize the RAM
    So since the timeout starts and finishes later, it works. This could be as trivial as the fact that reading/writing 48GB of RAM probably takes, say, 5 seconds if not longer. If the timeout is set to, say, 5.1 seconds because the board was only ever tested with 48GB, then adding any more memory will push it over the timeout threshold.
     
    I am, of course, just guessing here; I have not tried decompiling the BIOS code, nor do I have acces to the source for it. Even the documentation for the diagnostic codes is woefully incomplete; when 96GB POST fails, the last codes are b8, b9, cf, reset, and none of those codes are documented.
     
    I can completely relate to your disappointment and exhaustion from this motherboard - it really is a product that was designed for claiming bragging rights but turned out to be an embarrassment instead.
     
    I have yet to check verify whether 48GB/CPU success follow the CPU or the socket. If it follow the CPU then clearly, some CPUs are a little better and manage it within the timeout period. If it follows the socket then the motherboard plays the part as well, and in that case we are screwed without a BIOS patch to address the issue, which let's face it, we're not going to get considering the BIOS developers couldn't even get something as basic as the profile saving/loading to work completely properly.

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #3
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2014/01/07 08:47:59 (permalink)
    Just out of interest - has anybody actually tried quad-ranked 8GB RDIMMs on the SR-2? While this might necessitate reducing the RAM clock speed from 1600+ to 1333, most CPUs have issues with losing banks over 1333 anyway, so this is hardly an issue. I've looked through various forums and cannot find any mention of anyone having actually tried quad-ranked DIMMs, so I'm wondering if they might actually just work. Specifically:
     
    1) Single-ranked 6x8GB RDIMMs just works
    2) Single-ranked 12x8GB RDIMMs doesn't work
    3) Dual-ranked 12x8GB RDIMMs sometimes works (seemingly only on CPUs from later production runs)
     
    While this is in itself a bit bizzare (DR 6x8GB is electrically and electronically near-identical to SR 12x8GB), let's just take it at face value at the moment. On the face of it, the only apparent pattern is that the problem is primarily linked to the amount of memory per rank. Thus it stands to reason that QR 12x8GB might actually work on earlier CPUs.
     
    Hence my question - has anyone actually tried QR, or is the suggestion that it won't work coming purely from nobody having tested it?

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #4
    mpogr
    New Member
    • Total Posts : 34
    • Reward points : 0
    • Joined: 2007/01/26 01:55:44
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/01/07 22:47:20 (permalink)
    Mate, I'm over with this board. I have it working now with 48GB of memory at reasonable overclocked speed (3.6GHz) and don't want to waste any more time on fiddling with it. I use it as VMWare ESXi host running 10 VMs (one of them is OpenSolaris serving a large ZFS volume). I could definitely run at least twice as many VMs if I had 96GB, but I REALLY don't like the idea of spending more time (and money) on finding out how to do it.
    #5
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2014/01/11 13:36:29 (permalink)
    I've made some progress and I have some additional info.
    1) CPU
    I've switched my CPUs around, and it looks like both the socket and the CPU make a difference. Socket 0 seems to fare a little better than Socket 1, all things being equal. Still, the CPU makes more difference. My two CPUs are:
    1.1) Costa Rica 3124Bxxx batch - generally POSTs 48GB on it's own within 2-3 POST cycles
    1.2) Malay L037Bxxx batch - not as good - didn't seem to POST 48GB at all in socket 1, but manages it eventually in socket 0.
     
    2) Sockets
    For any individual CPU I have more luck POST-ing 48GB on socket 0 than on socket 1, but
     
    3) RAM
    I am using Crucial CT8G3ERSLD4160B modules (DDR3-1600 RDIMMs x4 Dual-Ranked) for 48GB/socket.
    I have yet to successfully POST 48GB of single-ranked modules per socket, but I haven't set tried using them with what I'm about to explain in section 4).
     
    4) What seems to boost the success rate and reduce the number of cycles before 48GB/socket posts is the following:
    1) Get your machine booted with 6 DIMMs:
    Memory Speed = DDR-1333
    MCH Strap = 1600MHz
    All memory timings set to "Auto"
    2) Go to memory timings and set ALL of them manually to what the "Auto" setting defaulted them to (number on the right for each setting).
    Alter the following:
    Command Rate = 2 (defaults to 1)
    Back-to-Back CAS Delay = 3 (defaults to 0)
    3) Go to voltage configuration, and set:
    CPU to 1.30V initial and eventual
    VTT to 1.35V initial and eventual
    CPU 0 DIMM / CPU 1 DIMM = 1.65V
    Note 1: All 3 of these settings are what "Auto" defaults them to, but disabling auto-detection seems to help with POST-ing 48GB/socket.
    Note 2: 1.65V is arguably ridiculously high for DIMMs designed for 1.35V - you may be able to get things working with a value lower than 1.65V set here, but nevertheless, that is what the default is as soon as you start overclocking.
    Note 3: Reducing memory speed selection to DDR-1066 causes the POST to fail very quickly, and loops endlessly every couple of seconds.
    With these settings I can post my Malay CPU in socket 0 with 48GB within 4-5 POST cycles. Once the machine POSTs with all the RAM once, there are no problems on soft-reboots, everything comes up immediately; it is only on a cold-boot that it takes a few POST cycles for the BIOS to come up.
     
    I can also confirm that after clearing the CMOS, the machine POSTs with 48GB/socket within a few cycles, as per mpogr's findings.
     
    This is what mostly works for me. Your mileage may vary depending on your CPUs.

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #6
    naimc
    New Member
    • Total Posts : 5
    • Reward points : 0
    • Joined: 2013/12/20 06:44:08
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/01/13 18:56:56 (permalink)
    Thank you  bygordan79 .
     
    I followed your steps and got my SR-2 with dual x5560's working with 96 GB of ram using 6 x 16GB Dual Rank DIMMS (Kingston KVR13LR9D4K3/48,  () 
     
    Excellent.
    #7
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2014/01/14 00:33:01 (permalink)
    Just out of interest - did you have any luck with getting it to post with 6x16 on a single socket? Also, do you have access to quad-ranked RAM you could try? The thing that bothers me is that a DR DIMM is electrically and electronically identical to 2xSR DIMMs on a single bank. Yet the former works and the latter does not. If the problem is RAM "density", rank count is used to reduce that, so bumping up the rank count to quad might help, even though it would necessitate down clocking the RAM.
     
    Also, with 6x16, does it "just POST" on a cold boot or does it require many cycles to get it to come up?

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #8
    mpogr
    New Member
    • Total Posts : 34
    • Reward points : 0
    • Joined: 2007/01/26 01:55:44
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/01/14 00:38:29 (permalink)
    Wow, 16GB modules working on SR-2 is already huge news, I think this is the first report of this kind! I'm really keen to know if it really works beyond just posting. Thanks!
    #9
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2014/01/14 01:45:51 (permalink)
    I'd also be interested to hear the batch number of the CPUs you used. It seems pretty clear by now that the chances of success with more recent CPUs are much higher.
     
    Also, I think the reason we haven't heard reports of success with 16GB DIMMs is because until recently they were prohibitively expensive, in the same way that 32GB ones still are. There is a good chance that 32GB ones quad-ranked ones would work, too, but I don't happen to have £2500 to spend on 6 of them for testing.

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #10
    naimc
    New Member
    • Total Posts : 5
    • Reward points : 0
    • Joined: 2013/12/20 06:44:08
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/01/14 05:50:19 (permalink)
     
    No, I did not try the 6x16 on single socket with single CPU but I will try that next time I test. ( IN a few weeks )
    No, don't have access to quad rank 16GB dims.
     
     
    gordan79
    Just out of interest - did you have any luck with getting it to post with 6x16 on a single socket? Also, do you have access to quad-ranked RAM you could try? The thing that bothers me is that a DR DIMM is electrically and electronically identical to 2xSR DIMMs on a single bank. Yet the former works and the latter does not. If the problem is RAM "density", rank count is used to reduce that, so bumping up the rank count to quad might help, even though it would necessitate down clocking the RAM.
     
    Also, with 6x16, does it "just POST" on a cold boot or does it require many cycles to get it to come up?




    #11
    stingr9137
    New Member
    • Total Posts : 15
    • Reward points : 0
    • Joined: 2014/01/03 17:40:35
    • Status: offline
    • Ribbons : 1
    Re: SR-2 - 96GB Secret Sauce 2014/02/07 12:55:14 (permalink)
    Naimc,  This is amazing news.  I've been doing an SR-2 build for the past month.  The original intent was to use 16GB sticks. 
     
    Any news on the use of 6x16 on a single socket.  I actually need more than 96GB aggregate across both processors so the answer to 6x16 on one socket is very useful to me.
     
    Any chance I could give you a call to compare notes in real-time?
    Thanks naimc.
    #12
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2014/02/07 13:01:54 (permalink)
    I think you are pushing your luck with 96GB already. It looks like 6x16GB has approximately the same success ratio as 12x8GB, and is likely to only work with newer Xeons, as the process improved over time, reducing current leakage and managing to initialize the memory despite SR-2's unsound initialization settings.
     
    What you might be able to get away with is using quad-ranked DIMMs since that reduces the load further, but if you fill up all the slots, that should cause the memory to get downclocked - and I am not sure I would trust the SR-2's BIOS to even implement that, considering EVGA never even tested it with anything other than dual-ranked 4GB DIMMs.

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #13
    stingr9137
    New Member
    • Total Posts : 15
    • Reward points : 0
    • Joined: 2014/01/03 17:40:35
    • Status: offline
    • Ribbons : 1
    Re: SR-2 - 96GB Secret Sauce 2014/02/07 13:39:25 (permalink)
    I'm wondering if Naimc's success has anything to do with 1.35v DIMMs.  I purchased KVR13R9D4K4/64 and installed.  It was the most frustrating experience ever.  I experienced the following:
     
    Success Scenario:
    1. Reset CMOS
    2. Board POST fine
    3. Receive initial bios message requesting user to enter bios or load defaults and continue
    4. Enter Bios
    5. Change NOTING
    6. Press 'esc' (do not save.  just exit)
    7. Boot process continues to windows
    8. All memory recognized.  Windows works fine.
    9. Repeat from step 1 every time you boot (so its essentially ridiculous since you cannot modify any settings in the bios)
     
    Failure Scenarios:
    Scenario 1)
    1. Reset CMOS
    2. Board POST fine
    3. Receive initial bios message requesting user to enter bios or load defaults and continue
    4. Enter Bios
    5. Change NOTHING (not even the date)
    6. Save and exit bios
    7. Board attempts to reboot
    8. Board never POSTs
    9. Reset CMOS to try something different
     
    Scenario 2)
    1. Reset CMOS
    2. Board POST fine
    3. Receive initial bios message requesting user to enter bios or load defaults and continue
    4. Enter Bios
    5. Change ANYTHING (even just the date)
    6. Save and exit bios
    7. Board attempts to reboot
    8. Board never POSTs
    9. Reset CMOS to try something different
     
    Scenario 3)
    1. Reset CMOS
    2. Board POST fine
    3. Receive initial bios message requesting user to enter bios or load defaults and continue
    4. Select Load Defaults and continue
    5. Board attempts to reboot
    6. Board never POSTs
    7. Reset CMOS to try something different
    #14
    stingr9137
    New Member
    • Total Posts : 15
    • Reward points : 0
    • Joined: 2014/01/03 17:40:35
    • Status: offline
    • Ribbons : 1
    Re: SR-2 - 96GB Secret Sauce 2014/02/07 13:56:17 (permalink)
    The very strange thing is that a scenario clearly exists where the board will work with 16gb RDIMMs.  Maybe manual setting the values would have fixed my problem.  Wish I still had the 16gb sticks to test.
    post edited by stingr9137 - 2014/02/07 20:33:16
    #15
    sam nelson
    iCX Member
    • Total Posts : 370
    • Reward points : 0
    • Joined: 2014/01/06 21:07:07
    • Status: offline
    • Ribbons : 9
    Re: SR-2 - 96GB Secret Sauce 2014/02/09 06:06:43 (permalink)
    yes u are right but can at times be a little bit of a pain . its kind of like 4way sli with 780 ti's and doing 4way with 780's the 780ti's will 4way with all the new drivers and game ready stuff np. you can do 4way with 780's but u have to go back to a holder set of driver . and pull some titan inf fill out of one and put thim in the fill . it work very well found it on the net the only problem is if u play a game online with them that updata's every time u play and are using win 8.1 it redoes your drivers to all new and drops u back to 3way all that is supported for them but they will runn 4way with the right software
     
    #16
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2014/02/09 07:05:17 (permalink)
    What does this have to do with getting 96GB of RAM to work on a SR-2??

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #17
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2014/02/15 03:48:57 (permalink)
    I just did a bit more testing and can confirm that 16GB x4 DR 1600MHz ECC RDIMMs do not work for me on the SR-2, even with all the above tricks applied to get it to POST. I have not managed to get it to POST with even a single DIMM in any of the slots. I haven't tried clearing the CMOS (couldn't be bothered to re-dial in all the settings since not everything gets saved in the profile).
     
    The board does, however, post every time (usually after a 2-3  diagnostic cycles) with 6x8GB x4 DR 1600MHz ECC RDIMMs on each CPU slot. From this I can only conclude that 12x8GB is a safer bet on the SR-2 than 6x16GB, all other things being equal.
     
    It also looks like 4GB/rank seems to be the safest option. 3x 8GB SR RDIMMs per CPU works, but 6x8GB per CPU does not. 6x8GB DR RDIMMs per CPU works most of the time.
     
    I can also now confirm that full 12x8GB DR RDIMMs works for me. It takes anything up to about 10 b8-b9-cf cycles to POST, but eventually it gets to b8-b9-d4-cf then b8-b9-b4-b5 and then it finishes POST-ing. My gut feeling is still that there is a watchdog timer being used during initialization and if the initialization time exceeds a timeout value for a particular step, it triggers a reset but once each step has completed initialization the value in the relevant registers sticks, so initialization takes a little less time, eventually leading to completing the POST. If this is the case, then it is a combination of 2 bugs (watchdog timeout being too short, and reset not clearing the registers that got initialized) partially cancelling each other out, leading to a successful POST with 96GB.
     
    So to summarize, to get 96GB working:
    1) Use retail Xeons, as late as possible. ES ones don't seem to work.
    2) Use DR rather than SR RDIMMs
    3) Use 8GB DIMMs, 16GB DR ones may work for some people but they didn't work for me.
    4) I only tried x4 RDIMMs in all cases
     
    I used these DIMMs from Crucial:
    CT8G3ERSLD4160B
     
    The results aren't perfect but if you can live with POST taking a minute or two it is workable
     
    Thinks that haven't been attempted yet as far as I can find:
    1) Using x8 rather than x4 RDIMMs
    2) Using QR RDIMMs
     
    Either of those options may or may not help. Using 6xQR RDIMMs shouldn't require downclocking to 1333MHz. Using 12xQR DIMMs will.
    post edited by gordan79 - 2014/02/24 00:43:34

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #18
    rjohnson11
    EVGA Forum Moderator
    • Total Posts : 76486
    • Reward points : 0
    • Joined: 2004/10/05 12:44:35
    • Location: Europe
    • Status: online
    • Ribbons : 70
    Re: SR-2 - 96GB Secret Sauce 2014/02/24 01:02:04 (permalink)
    sam nelson
    yes u are right but can at times be a little bit of a pain . its kind of like 4way sli with 780 ti's and doing 4way with 780's the 780ti's will 4way with all the new drivers and game ready stuff np. you can do 4way with 780's but u have to go back to a holder set of driver . and pull some titan inf fill out of one and put thim in the fill . it work very well found it on the net the only problem is if u play a game online with them that updata's every time u play and are using win 8.1 it redoes your drivers to all new and drops u back to 3way all that is supported for them but they will runn 4way with the right software
     


    Please stay on topic of the original OP

    Specs: AMD Ryzen 9 3950X,  Corsair MP600 M.2, 64GB Corsair RGB Dominator,  ASUS Crosshair VIII Hero,  Corsair 1000D, EVGA RTX 2070 FTW 3

    Associate Code: H5U80QBH6BH0AXF



    #19
    mcllz
    New Member
    • Total Posts : 7
    • Reward points : 0
    • Joined: 2014/02/25 23:54:23
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/03/02 17:12:02 (permalink)
    I am ready to build the 2 cpus system,but 2*x5675es,if with 12*4gb reg ecc ram,can it overclock?any advic?
    #20
    tived
    iCX Member
    • Total Posts : 263
    • Reward points : 0
    • Joined: 2010/09/06 23:12:46
    • Location: Sydney : NSW : Australia
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/03/02 18:12:38 (permalink)
    mcllz,
     
    try and stay on the subject, you are clearly not looking to run with 96gb of ram.
     
    Yes, you can run with 12x 4gb and yes it can be over clocked - but you will probably have more luck with 6x8gb
     
    please use the search function - I don't mean to be rude, but you need to make a little effort here to get things up and running. there used to be some templates for overclocking here. do a search
    all the best an enjoy your new rig
     
    Henrik

    Henrik
    - A Dane Down Under 
    Current systems: 
    EVGA Classified SR-2  Lian Li PC-V2120 Black, Antec 1200 PSU,
    2x X5650 (stock/4.2xxGhz), CPU Cooling: Noctua NH-D14 - Ram: (48gb) 6x 8Gb Kingston ECC 1333 KVR1333D3D4R9S/8GI - Disks & controllers:  Boot (Areca 1882ix-24): 8R0 SAMSUNG 830 128GB - 2x IBM M1015/Scratch disk: 2R0(6R0) INTEL 520 120GB's RAID-00 (12 SSD'S) - GPU: Asus GTX-580 - Monitors: NEC 2690v2 & Dell 2405 - Profiler X-rite: Eye-One Pro
    #21
    mcllz
    New Member
    • Total Posts : 7
    • Reward points : 0
    • Joined: 2014/02/25 23:54:23
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/03/02 18:22:15 (permalink)
    I want to try 96gb ram,but I do care if it can  overclock or if it have booting issues,Yes,my cpus is here,even 4 x5675 es is here,but I have no ecc ram,I have 12*8gb non ecc rams,but obviously,it cant work,so I hope I can get help.which reg ecc rams I need to buy?,I am sorry,I tried to do search,but unfortunately,my Enlish is poor,so if you can give me some advice,I will appreciate you very much,thank you again,
     
     
    tived
    mcllz,
     
    try and stay on the subject, you are clearly not looking to run with 96gb of ram.
     
    Yes, you can run with 12x 4gb and yes it can be over clocked - but you will probably have more luck with 6x8gb
     
    please use the search function - I don't mean to be rude, but you need to make a little effort here to get things up and running. there used to be some templates for overclocking here. do a search
    all the best an enjoy your new rig
     
    Henrik





    #22
    mcllz
    New Member
    • Total Posts : 7
    • Reward points : 0
    • Joined: 2014/02/25 23:54:23
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/03/02 18:27:48 (permalink)
    I mean if I want to overclock it,even 4.5ghz,which 8gb rams can work?12*8 is the best,dual-rank ddr3 1600?
     
    #23
    tived
    iCX Member
    • Total Posts : 263
    • Reward points : 0
    • Joined: 2010/09/06 23:12:46
    • Location: Sydney : NSW : Australia
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/03/03 20:07:05 (permalink)
    Hi,
    I had no luck over locking with 12x 8gb ecc 1333 Kingston on my X5650's
    However others have been able to do so with Crucial ram but they have also had faster processors X5680 which some seem to think will make a difference.

    You can try with your 12x 8gb non-ecc they should work but if they will over lock is the luck of the draw 😄

    Henrik

    Ps: English is my second language too ☺️

    Henrik
    - A Dane Down Under 
    Current systems: 
    EVGA Classified SR-2  Lian Li PC-V2120 Black, Antec 1200 PSU,
    2x X5650 (stock/4.2xxGhz), CPU Cooling: Noctua NH-D14 - Ram: (48gb) 6x 8Gb Kingston ECC 1333 KVR1333D3D4R9S/8GI - Disks & controllers:  Boot (Areca 1882ix-24): 8R0 SAMSUNG 830 128GB - 2x IBM M1015/Scratch disk: 2R0(6R0) INTEL 520 120GB's RAID-00 (12 SSD'S) - GPU: Asus GTX-580 - Monitors: NEC 2690v2 & Dell 2405 - Profiler X-rite: Eye-One Pro
    #24
    mcllz
    New Member
    • Total Posts : 7
    • Reward points : 0
    • Joined: 2014/02/25 23:54:23
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/03/03 23:54:15 (permalink)
    tived
    Hi,
    I had no luck over locking with 12x 8gb ecc 1333 Kingston on my X5650's
    However others have been able to do so with Crucial ram but they have also had faster processors X5680 which some seem to think will make a difference.

    You can try with your 12x 8gb non-ecc they should work but if they will over lock is the luck of the draw 😄

    Henrik

    Ps: English is my second language too ☺️

    thank you for your message,but I do want to know if x5675 es(a0) can work with 8gb*12(samsung 1.35v ecc reg),any one has had a try?
    or any one has tried es cpus with 12*8gb dual-rank rams and it can work perfectly?
    #25
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2014/03/05 10:14:05 (permalink)
    4.5GHz? Dream on. If you get to 4GHz stably (including TurboBoost) count yourself very lucky. Don't believe the fisherman's tales.
     
    And did I read that right that your Xeons are ES Xeons? Abandon all hope of them picking up all your memory banks at more than 1333MHz RAM speed. In reality, that means you need to achieve either 166 bclk (with 1066 multiplier, relatively easily achieved with 48GB of RAM) or 222 bclk (not going to happen, don't even bother trying). In practice this means you'll be running with the 800 multiplier and actual memory clock speed substantially below 1333MHz.
     
    Nobody has ever managed to get ES Xeons to post with 12x8GB. Even early retail Xeons struggle to post with 12x8. 6x8GB works fine, however, as long as you don't go above 1333MHz on memory speed.
     
    Even with later retails Xeons, OC-ing is problematic. I am running 2x X5650 with 12x8GB with the methods mentioned above, and I was regularly getting correctable memory errors on CPU#1,Channel#2,DIMM#0 (i.e. first DIMM of the top channel on the second CPU). The error currently follows the slot rather than the DIMM (haven't tried swapping CPUs, the CPU in the second socket always seems a little more problematic with POST-ing with 6x8GB than the CPU in the 1st socket). This is also the memory bank that intermittently goes missing and doesn't get detected during POST on some boots when OC-ing. I have just dialed everything back to 133bclk (was running 166 bclk) to see if the error goes away.
     
    Unfortunately, troubleshooting this is quite difficult because getting the whole thing to POST with 96GB is difficult enough. For example, if I run:
    bclk: 166
    RAM: 1333 (1666 actual)
    MCH strap: 1600
    it generally posts (sometimes with 80GB showing, sometimes with all 96GB showing)
    If I reduce the RAM to 1066 (1333 actual with 166 bclk), it doesn't POST at all. It gets stuck in the error state withing a couple of POST attempts. So there is another interaction with something else other than pure memory clock speed going on that gets out of whack when you reduce the RAM speed. I suspect the uncore speed plays a part, but this will take some more experimentation to figure out.
     
    I have also established a few other useful things:
    1) vDIMM makes no difference - just set it to whatever your DIMMs are rated at (usually 1.35V). There is nothing to be gained from boosting this up to 1.65V that the "auto" setting defaults to when OC-ing, unless you are actually manging to run your DIMMs at faster speed than they are rated for (which you most likely won't succeed with 6 let alone 12 8GB DIMMs, especially since you can get 1800MHz DIMMs)
    2) In fact - most voltages make no difference to POST-ing with 96GB without OC-ing. I currently have reliable posting with all my voltages hard-set to defaults:
    vCore: 1.0875
    VTT: 1.20
    vDIMM: 1.35
    IOH: 1.100
     
    with all the multipliers hard-set to default:
    bclk: 133
    CPU multiplier: x20
    QPI: 6.4GT
    RAM: 1333
    MCH strap: 1333
    Uncore: x20
     
    With the DIMMs I listed above (Crucial DDR3-1600 ECC RDIMM x4 DR 1.35V) this POSTs within 10 seconds every time. I'm going to leave it running for a week and give it a proper workout to make sure the correctable RAM error has gone away. If it has, I'll start tuning my way back up to 166 bclk.
     
    So to summarize:
    1) You are not very likely to achieve 100% stability beyond 4GHz including turbo boost. With your X5675s that means 166 bclk + TB. I very much doubt you will achieve this with ES Xeons.
    Note: I define "stable" as no errors during OCCT + some of my extra proprietary stability testing based on md5sum and sha512sum computations) for about a week. 24 hours each of OCCT small and large standard and linpack tests, 4 days in total, plus 24 hours of sha512sum on a file small enough to fit into CPU caches for CPU core testing plus 24 hours of md5sum on large randomly generated files big enough to fill up the whole RAM to properly exercise the memory controllers.
    2) You are very unlikely to get 96GB to POST with ES Xeons.
    3) Even with using 48GB of RAM with ES Xeons expect that some of your RAM won't get detected at over 1333MHz RAM speed. Remember - X56xx Xeons were only ever designed and rated for 1333MHz RAM speed in the first place.

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #26
    mcllz
    New Member
    • Total Posts : 7
    • Reward points : 0
    • Joined: 2014/02/25 23:54:23
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/03/05 23:04:55 (permalink)
    if with 48g ram,what is better if I want to oc high cpu frequency。12*4gb or 6*8gb?what is your advice?thank you for your message
    gordan79
    4.5GHz? Dream on. If you get to 4GHz stably (including TurboBoost) count yourself very lucky. Don't believe the fisherman's tales.
     
    And did I read that right that your Xeons are ES Xeons? Abandon all hope of them picking up all your memory banks at more than 1333MHz RAM speed. In reality, that means you need to achieve either 166 bclk (with 1066 multiplier, relatively easily achieved with 48GB of RAM) or 222 bclk (not going to happen, don't even bother trying). In practice this means you'll be running with the 800 multiplier and actual memory clock speed substantially below 1333MHz.
     
    Nobody has ever managed to get ES Xeons to post with 12x8GB. Even early retail Xeons struggle to post with 12x8. 6x8GB works fine, however, as long as you don't go above 1333MHz on memory speed.
     
    Even with later retails Xeons, OC-ing is problematic. I am running 2x X5650 with 12x8GB with the methods mentioned above, and I was regularly getting correctable memory errors on CPU#1,Channel#2,DIMM#0 (i.e. first DIMM of the top channel on the second CPU). The error currently follows the slot rather than the DIMM (haven't tried swapping CPUs, the CPU in the second socket always seems a little more problematic with POST-ing with 6x8GB than the CPU in the 1st socket). This is also the memory bank that intermittently goes missing and doesn't get detected during POST on some boots when OC-ing. I have just dialed everything back to 133bclk (was running 166 bclk) to see if the error goes away.
     
    Unfortunately, troubleshooting this is quite difficult because getting the whole thing to POST with 96GB is difficult enough. For example, if I run:
    bclk: 166
    RAM: 1333 (1666 actual)
    MCH strap: 1600
    it generally posts (sometimes with 80GB showing, sometimes with all 96GB showing)
    If I reduce the RAM to 1066 (1333 actual with 166 bclk), it doesn't POST at all. It gets stuck in the error state withing a couple of POST attempts. So there is another interaction with something else other than pure memory clock speed going on that gets out of whack when you reduce the RAM speed. I suspect the uncore speed plays a part, but this will take some more experimentation to figure out.
     
    I have also established a few other useful things:
    1) vDIMM makes no difference - just set it to whatever your DIMMs are rated at (usually 1.35V). There is nothing to be gained from boosting this up to 1.65V that the "auto" setting defaults to when OC-ing, unless you are actually manging to run your DIMMs at faster speed than they are rated for (which you most likely won't succeed with 6 let alone 12 8GB DIMMs, especially since you can get 1800MHz DIMMs)
    2) In fact - most voltages make no difference to POST-ing with 96GB without OC-ing. I currently have reliable posting with all my voltages hard-set to defaults:
    vCore: 1.0875
    VTT: 1.20
    vDIMM: 1.35
    IOH: 1.100
     
    with all the multipliers hard-set to default:
    bclk: 133
    CPU multiplier: x20
    QPI: 6.4GT
    RAM: 1333
    MCH strap: 1333
    Uncore: x20
     
    With the DIMMs I listed above (Crucial DDR3-1600 ECC RDIMM x4 DR 1.35V) this POSTs within 10 seconds every time. I'm going to leave it running for a week and give it a proper workout to make sure the correctable RAM error has gone away. If it has, I'll start tuning my way back up to 166 bclk.
     
    So to summarize:
    1) You are not very likely to achieve 100% stability beyond 4GHz including turbo boost. With your X5675s that means 166 bclk + TB. I very much doubt you will achieve this with ES Xeons.
    Note: I define "stable" as no errors during OCCT + some of my extra proprietary stability testing based on md5sum and sha512sum computations) for about a week. 24 hours each of OCCT small and large standard and linpack tests, 4 days in total, plus 24 hours of sha512sum on a file small enough to fit into CPU caches for CPU core testing plus 24 hours of md5sum on large randomly generated files big enough to fill up the whole RAM to properly exercise the memory controllers.
    2) You are very unlikely to get 96GB to POST with ES Xeons.
    3) Even with using 48GB of RAM with ES Xeons expect that some of your RAM won't get detected at over 1333MHz RAM speed. Remember - X56xx Xeons were only ever designed and rated for 1333MHz RAM speed in the first place.




    #27
    gordan79
    SSC Member
    • Total Posts : 532
    • Reward points : 0
    • Joined: 2013/01/27 00:17:36
    • Status: offline
    • Ribbons : 3
    Re: SR-2 - 96GB Secret Sauce 2014/03/05 23:27:04 (permalink)
    I don't think it'll make much difference. 6x8 dual-rank will be virtually identical electrically and electronically as 12x4 single-rank DIMMs. Having said that, I also use 6x8GB single-rank (12x8 SR won't work) with no issues. Just make sure you keep your RAM at <= 1333MHz actual speed and it should work fine. My advice would be to aim for 166bclk with 1066 memory speed set in the BIOS which will give you 1333 actual RAM speed, and 166 bclk with boost enabled should get you to around 4GHz if your CPUs can actually handle it (which isn't that likely with ES chips - if it's unstable, disable TB). Don't push vCore past 1.30V or VTT past 1.35V, regardless of how good your cooling is as there have been reports of CPU damage.

    Supermicro X8DTH-6, 2x X5690
    Crucial 12x 8GB x4 DR 1.35V DDR3-1600 ECC RDIMMs (96GB)
    3x GTX 1080Ti
    Triple-Seat Virtualized With VGA Passthrough (KVM)
    #28
    mcllz
    New Member
    • Total Posts : 7
    • Reward points : 0
    • Joined: 2014/02/25 23:54:23
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/03/05 23:36:54 (permalink)
    gordan79
    I don't think it'll make much difference. 6x8 dual-rank will be virtually identical electrically and electronically as 12x4 single-rank DIMMs. Having said that, I also use 6x8GB single-rank (12x8 SR won't work) with no issues. Just make sure you keep your RAM at <= 1333MHz actual speed and it should work fine. My advice would be to aim for 166bclk with 1066 memory speed set in the BIOS which will give you 1333 actual RAM speed, and 166 bclk with boost enabled should get you to around 4GHz if your CPUs can actually handle it (which isn't that likely with ES chips - if it's unstable, disable TB). Don't push vCore past 1.30V or VTT past 1.35V, regardless of how good your cooling is as there have been reports of CPU damage.


    I have 6*8gb non ecc rams,do you think it will works as ecc reg rams?I am sorry for so many questions,but my MB is now on the way.


    #29
    mcllz
    New Member
    • Total Posts : 7
    • Reward points : 0
    • Joined: 2014/02/25 23:54:23
    • Status: offline
    • Ribbons : 0
    Re: SR-2 - 96GB Secret Sauce 2014/03/05 23:38:43 (permalink)
    I searched many posts,found only one member said it will work with 6*8 non ecc ram,but I cant confirm with him
    #30
    Page: 1234 > Showing page 1 of 4
    Jump to:
  • Back to Mobile