I've solved the acpi interrupt storm I used to get with a grub argument passing of noapic and acpi=off (well not a real solution, a real solution would probably be a kernel recompile) but if anyone had any experiences with SR2's and Xen and this interrupt storm, some pointers would be helpful thanks
 
 

Try booting the kernel with pcie_aspm=off
 

Good luck, and be prepared for a whole world of pain and maintaining your own custom build of Xen. IOMMU on the SR-2 is badly broken due to the NF200 bridges.
 
It's doable with a few bodges - my bodge results in 2.5GB of RAM going missing from each VM you run, so if you give a VM 8GB only 5.5GB will be usable. But with 96GB of RAM in the machine and two 16GB VMs I can just about live with that.
 
The problem is that when you assign the VM more RAM than the bottom of the used PCI IOMEM range the data the VM writes to what it thinks is memory ends up being written to the PCI aperture, which both crashes the card using that memory and usually the host as well. If you are unlucky and it overwrites IOMEM belonging to a disk controller it will trash the contents of the disks, too. It is quite a dangerous and nasty hardware design bug of the NF200 bridges. My solution is to set up the guest memory with the memory hole in the guest being large enough to cover all of the physical PCI IOMEM range, and by luck more than anything, where Xen's Seabios and HVMLoader decide to map the GPU's IOMEM in the VM happens to not clash with any of the physical IOMEM ranges. It's a horrible bodge, but it works for me.
 
You will then need some modified Nvidia cards; I recommend a GTX480 BIOS soft-modded strap to Quadro 6000, a GTX680 hard-modded to a Tesla K10 by removing one resistor off the back of the card, or a GTX780Ti hard-modded to a Quadro K6000 by soldering a 33K 1206 resistor across two of the EEPROM pins. Make sure you don't get a Gainward card, they can be problematic with VGA passthrough.
 
To summarize - it can be made to work, but be prepared to put in a non-trivial amount of work into it. I'm happy to share details and patches so you won't have to do the hardest part of the bleeding edge trail blazing, but if you aren't comfortable with building your own packages and BIOS hex-editing safe yourself the trouble and get a different motherboard that is known to work properly in the kind of setup we are discussing here.

Your system is indeed nearly identical to mine. I wouldn't bother clocking the CPUs down if you can keep them cool. The main stability bottleneck is bclk. I find mine get up to 90C under full load with 3 GPUs and 15 disks also running flat out (e.g. CudaMiner on GPUs with disks scrubbing, and miners on the CPUs). Your temperatured may differ.
 
If you won't be using GPU passthrough you shouldn't have any problems at all. I use Xen because it is the only one with even a hope of running GPU passthrough on this motherboard. If that isn't a factor for your usage, use whatever is best supported and documented on your distro. Disable IOMMU and it should all work just fine. You could even use VMware Player 5.x (6.x 3D acceleration is broken) if you only plan to do casual gaming at <= 1080p.

I only just noticed you plan to run a database load in a VM. You should be aware that you are likely to see a throughput drop at full saturation of about 40% from running virtualized. Probably not an issue if it's just a development/testing/play machine, though.
 
Prime95 is useless at stability checking. Use OCCT and minerd, it will show up issues much more reliably.
 
I am cursing most of the recent, pointless changes in distros, both in Grub2 and the switch to systemd. Complete waste of time for no meaningful benefit and a massive increase in complexity and decrease in transparency.

I did some testing, see here:
http://www.altechnative.net/2012/08/04/virtual-performance-part-1-vmware/
 
The hit on database loads will be worse because the disk I/O translation is expensive even when paravirtualized. You might avoid some of it if you pass through the SATA/SAS controller using PCI passthrough to the guest, but then you are back in the situation where you need working IOMMU, as per the GPU passthrough case described above. I run paravirtualized I/O drivers on HVM Windows guests, and the overhead is very much non-trivial.

The thread at the end of your post was informative. I've never thought about it really but I used to run esxi and then just virtualize a single vm and always gave it a few cpus less than max, hah what a waste.
 
Given the hassle I might just baremetal the sr2 and use the supermicro system for zfs

Yes a Zvol would be better as I can set individual settings on it like compression and dedup.
 
Regarding the two machines: That's what I'm debating. On one hand I hate the idea of leaving performance on the table in the name of consolidation. But it's not a production server so... And I can sell the supermicro box and have more cash for other things. But since this project really started to be a learning experience, perhaps wading through some infiniband / iSCSI issues might be a good learning process.

This is true, especially with the wealth of knowledge from people who have done it before.
 
This is going to be a fun weekend.

The problem is that there are so many pitfalls with PCI pass through. That is the reason why there is only a handful of workstations certified to work with VGA passthrough.
 
I've been meaning to write it all up for months, but I haven't gotten around to it. Partly because the Xen patch I had to write isn't even remotely production ready or suitable for public consumption. There are issues with it, for example, the memory that goes missing in domU.

I did have a question for you though seeing as you have quite a bit of hardware knowledge. Is the option rom space contained within the eeprom for the bios? If so is it possible to get a larger eeprom and port the bios to the new chip?

I didn't mean to derail the thread. I haven't actually thought about upgrading my setup in a wile because it actually does exactly what it is supposed to. Back in the day though it seemed to be a problem to get 4 - 4gig gpu's along with a raid controller to not run into option rom limitations. Jacob seemed to have eluded to the point that they ran out of bios space to fix this and a few of the other bios issues. I am FAR from having the patience to pursue learning how to rewrite bioses though and most of the people who were doing that sort of thing have long since moved on to newer platforms. I could be completely chasing the wrong animal but it seems that some of the incompatibility issues with raid cards and the newer pcie ssds could benefit from extra free option rom. probably the whole bios would need to be ported (I am guessing here ) from 32bit or 16 bit to 64 bit to take advantage though.

pcie_aspm=off didn't work for booting xen but no biggie. I decided to try out kvm + virtio and for a total noob it's working great. Virt-manager is a slick gui. I'll come back to xen should kvm + virtio prove awful

i added it to grub but it could have been a typo on my end. I'll circle back to it when my dinking around with kvm is done

All I can say is that it works for me with 3.9.9 kernel and pciehp module removed/blacklisted. Unfortunately pciehp is no longer a module in later kernels, but that is what pcie_aspm=off is supposed to work around.