gordan79
... There is a good chance you have a DIMM or CPU that isn't quite properly seated. If you decide to use ESXi, you should be aware that it is "headless", so you will need another machine to use as a workstation....
Or to put it another way, the SR-2 is a very poor choice of motherboard for virtualization. I have my setup working quite stably, but it took me weeks to figure out and work around the bugs.

gordan79
One thing that will break spectacularly is PCI passthrough of devices with memory apertures. There is a very serious bug in the NF200 PCIE bridges that causes them to bypass the IOMMU on PCIE DMA transfers. Most SAS controllers will cause a lock-up or just won't work at all with the IOMMU enabled. If you ever passthrough GPUs to your VMs you will have to use special workarounds to prevent the VM memory from overlapping the PCI IOMEM regions or the whole machine will crash as soon as the VM uses virtual memory space that overlaps with the physical IOMEM regions on the host.
 
As you can see from my signature, I managed to get this working reliable, but in hindsight, the weeks I spent figuring out the bug and working around it were worth far more than the cost difference to a good server board and top of the line Xeons to compensate for the lack of OC-ing features.

gordan79

roswellian2002

gordan79
One thing that will break spectacularly is PCI passthrough of devices with memory apertures. There is a very serious bug in the NF200 PCIE bridges that causes them to bypass the IOMMU on PCIE DMA transfers. Most SAS controllers will cause a lock-up or just won't work at all with the IOMMU enabled. If you ever passthrough GPUs to your VMs you will have to use special workarounds to prevent the VM memory from overlapping the PCI IOMEM regions or the whole machine will crash as soon as the VM uses virtual memory space that overlaps with the physical IOMEM regions on the host.
 
As you can see from my signature, I managed to get this working reliable, but in hindsight, the weeks I spent figuring out the bug and working around it were worth far more than the cost difference to a good server board and top of the line Xeons to compensate for the lack of OC-ing features.

I'm really happy to come across this post. I originally planned to get a used SR-2 for GPU passthrough on ESXi...After reading the posts all, it seems to be impossible. I still have one more question...what if I assign > 4G memory for a VM, will passthrough work? Thanks.

 
Short answer:
Don't get an SR-2.
 
Long answer:
IIRC ESXi doesn't work with GPU passthrough unless the IOMMU supports ACS. NF200 bridges don't, and thus (at least recent versions of) ESXi will flat out not do it, period, regardless of any workarounds. Xen and KVM can be persuaded to ignore the lack of ACS (and the security impact of doing so).
 
Without any additional workarounds, you should be able to get PCI passthrough working with VMs allocated (4GB - PCI memory selected in BIOS). So if you set it in BIOS to 3G (max), giving a VM more than 1GB is risky if you are using PCI passthrough. If you set the PCI memory gap to 1GB you may be able to get away with giving the VMs up to 3GB. Other issues such as various lock-ups dependant on what other hardware you have may manifest regardless.
 
With recent Xen and KVM you can work around the problem by using a QEMU option to limit low memory allocation (e.g. to 1GB) to make sure the VM's memory map never includes any PCI aperture address ranges.
 
Lack of ACS still has security implications, though.
 
So with ESXi you have no hope because the hardware is to broken. With Xen or KVM you can get away with it, but make sure you understand the ramifications and limitations.