I saw instructions on how to install vmware workstation on ubuntu... so i guess i dont need windows.

Maybe i will just install esxi host and have my vc implimented on my host ina nested environment.

gordan79
... There is a good chance you have a DIMM or CPU that isn't quite properly seated. If you decide to use ESXi, you should be aware that it is "headless", so you will need another machine to use as a workstation....
Or to put it another way, the SR-2 is a very poor choice of motherboard for virtualization. I have my setup working quite stably, but it took me weeks to figure out and work around the bugs.

Other than the initial start, this seems to work fully.
I doubt that anything is seated wrong. I have full functionality with ubuntu with 94.4gb of usable memory... assuming that the rest is lost in conversion. I currently am running workstation 12 on ubuntu. I have 6 hosts implimented and multiple windows machines on each host with FT implemented.
 
As for the stability of the board... no issue running this for a few days on full load... just that issue when resetting. I tried delaying the post so that the memory can initialize.
 

Something like that

It looks like spaghetti. And as you can see my ram is being almost maxed out from my virtual machines... Currently running almost a week without failure.
 

I think I will hang on to this one...
So far... other than the reset, I havn't had any problems or crash.
All of my VMs are in good standing, so it is working for what I need it for.
No need to sink money into something else.
I am running multiple windows systems and different flavors of unix. I never power the system down anywhos... lol

wow... how did you get the reset to work without clearing the cmos?
I just never turn this thing off. Still no crash yet.

Great!

gordan79
One thing that will break spectacularly is PCI passthrough of devices with memory apertures. There is a very serious bug in the NF200 PCIE bridges that causes them to bypass the IOMMU on PCIE DMA transfers. Most SAS controllers will cause a lock-up or just won't work at all with the IOMMU enabled. If you ever passthrough GPUs to your VMs you will have to use special workarounds to prevent the VM memory from overlapping the PCI IOMEM regions or the whole machine will crash as soon as the VM uses virtual memory space that overlaps with the physical IOMEM regions on the host.
 
As you can see from my signature, I managed to get this working reliable, but in hindsight, the weeks I spent figuring out the bug and working around it were worth far more than the cost difference to a good server board and top of the line Xeons to compensate for the lack of OC-ing features.

I'm really happy to come across this post. I originally planned to get a used SR-2 for GPU passthrough on ESXi...After reading the posts all, it seems to be impossible. I still have one more question...what if I assign > 4G memory for a VM, will passthrough work? Thanks.

gordan79

roswellian2002

gordan79
One thing that will break spectacularly is PCI passthrough of devices with memory apertures. There is a very serious bug in the NF200 PCIE bridges that causes them to bypass the IOMMU on PCIE DMA transfers. Most SAS controllers will cause a lock-up or just won't work at all with the IOMMU enabled. If you ever passthrough GPUs to your VMs you will have to use special workarounds to prevent the VM memory from overlapping the PCI IOMEM regions or the whole machine will crash as soon as the VM uses virtual memory space that overlaps with the physical IOMEM regions on the host.
 
As you can see from my signature, I managed to get this working reliable, but in hindsight, the weeks I spent figuring out the bug and working around it were worth far more than the cost difference to a good server board and top of the line Xeons to compensate for the lack of OC-ing features.

I'm really happy to come across this post. I originally planned to get a used SR-2 for GPU passthrough on ESXi...After reading the posts all, it seems to be impossible. I still have one more question...what if I assign > 4G memory for a VM, will passthrough work? Thanks.

 
Short answer:
Don't get an SR-2.
 
Long answer:
IIRC ESXi doesn't work with GPU passthrough unless the IOMMU supports ACS. NF200 bridges don't, and thus (at least recent versions of) ESXi will flat out not do it, period, regardless of any workarounds. Xen and KVM can be persuaded to ignore the lack of ACS (and the security impact of doing so).
 
Without any additional workarounds, you should be able to get PCI passthrough working with VMs allocated (4GB - PCI memory selected in BIOS). So if you set it in BIOS to 3G (max), giving a VM more than 1GB is risky if you are using PCI passthrough. If you set the PCI memory gap to 1GB you may be able to get away with giving the VMs up to 3GB. Other issues such as various lock-ups dependant on what other hardware you have may manifest regardless.
 
With recent Xen and KVM you can work around the problem by using a QEMU option to limit low memory allocation (e.g. to 1GB) to make sure the VM's memory map never includes any PCI aperture address ranges.
 
Lack of ACS still has security implications, though.
 
So with ESXi you have no hope because the hardware is to broken. With Xen or KVM you can get away with it, but make sure you understand the ramifications and limitations.

Thanks a lot, gordon! That's really helpful. I'm moving to supermicro 2011 platform now.

takuhari
I was able to boot into ubuntu 16.4 with 96gb r2x4 hynix... windows will give me a bsod... any reset will boot loop and not post. Like earlier people, the start after the cmos clear works fine. Ubuntu runs good with it and it detects. I have e6520s so not the highend cpus but does have the 6 cores.

I am running at 1333mhz
Bios and ubuntu detects all 96gb

Did you ever get Win7 booting?
 
I am similarly late to this 96GB party, and I am afraid I've broke all the rules. I have a 2x X5650's with one being an ES (CPU1) and the second a retail SLBV3 (CPU0). The ES is old enough that it won't even POST if it is the primary CPU.
 
I also went and purchased, foolishly without googling, two 48GB (3x16GB) kits of Crucial CT3K16G3ERSLD4160B ECC non-REG thinking I could swap these in and be off to the kitchen all before the tea kettle whistles. That didn't happen of course.
 
However, I can POST with no problems! So score one for janky ES CPU. Unfortunately, Windows 7 bluescreens faster than I can read it and if I put my old RAM back in, no blue screen. Did you (or anyone else here) ever get a Windows 7 install up and running? I am going to try setting 2T, and maybe downclock to 1333, and see if that makes a difference when I have time, but I thought I would post here in case anyone happens to have an idea. Obviously Win7 is not the problem, but perhaps someone has an idea on how the ol' SR2 is screwing me this time. 

I also see a cheap cheap kit of 6x8 1333 ECC REG that I might switch to if need be. However, since I can POST, I am not convinced that will be the solution.
 

96GB DDR3 ECC REG 1333mhz overclocked to 1800mhz 10-10-10-28 and works great.
 

No problems
The memory is:
 
And the CPU is 2x xeon x5660 Works perfect.
going to test the 2x x5690 cpus later next week on the same mobo
A got x5645-x5650-x5660-x5670-x5680-x5690 so no problems :-)
 
When I opened A58 BIOS in AMIBCP 3.46 at Frequency/Voltage Control -> Memory Configuration I see hidden menu Memory Mode that can have 4 values: 00 (default), 01, 02 and 03. Who knows what it means and can it help in installation more than 48 Gb memory?
NO.
 

SR-2 works with 12x8Gb DIMMs Samsung M393B1K70DH0-YH9 and 2xX5690. Overclocking was not tried.

Dual xeon x5690, 96GB, 4550Mhz,  1,35v
So fast
 

So, yep, total noob here. I need some help as to how to un-grey the Memory Frequency Setting in the BIOS so that I can apply Robbanswe's settings. Btw, if anyone could walk me through the whole memory settings process, it would be a great blessing. I do not even know if, to modify the Memory Frequency Settings to 1800mhz as suggested by Robbanswe, one needs to modify any settings on the CPU side, i.e. clock speed or whatever. So, any help appreciated.