I'm thinking about setting up an X58 system with a Xeon X5670/87, and a 256GB SSD like this. Anyone know what kind of speed it would get on PCIe 2.0?

You'd get PCIE 2.0 X4 speeds, exactly half of the bandwidth available to a PCIE 3.0 X4 slot.
 
PCIE 2.0 has 500 MB/s of bandwidth per channel (X1).
PCIE 3.0 has 1000 MB/s of bandwidth per channel (X1).
 
PCIE 2.0 X4 has 2000 MB/s of bandwidth available, but version 2.0 has more overhead than 3.0 - so you won't see the full 2000 MB/s (just guessing, but 1500 MB/s sounds about right).
 
The difference in game load times and Windows load times from a standard, good quality, SATA III SSD to a high performance PCIE NVME SSD? ZERO, ZIP, ZILCH. You get faster file transfer speeds, and if you run a database server - very fast database access at high queue depths.
 
http://techreport.com/review/29221/samsung-950-pro-512gb-ssd-reviewed/4 
http://www.pcgamer.com/samsung-ssd-950-pro-review/ 
 
EDIT: And for the record, I have a Samsung 950 Pro 512GB NVME in a PCIE 3.0 expansion card, and I can attest to their findings. And yes, file transfers from it to my 8 drive RAID 6 array are very fast now (my game load times and my windows boot times are, sadly, the same).

Hmm, maybe it wouldn't be worth it for me then in light of that. Something to think about anyways.
 
Thank you!

A bit late to the party, but I'm installing a generic PCIe x4 NVME adapter card and a Samsung 970 EVO Plus into my EVGA X58 E762 tonight. Will run some real world tests on it. Not expecting miracles, but it's gotta be better than legacy platter hard drives by a mile. Will check back in on the thread once I have some numbers and a pic or two to post.

Brad_Hawthorne
A bit late to the party, but I'm installing a generic PCIe x4 NVME adapter card and a Samsung 970 EVO Plus into my EVGA X58 E762 tonight. Will run some real world tests on it. Not expecting miracles, but it's gotta be better than legacy platter hard drives by a mile. Will check back in on the thread once I have some numbers and a pic or two to post.

 
Almost 5.5 years Brad, lol.  
 
Hopefully your upgrade goes smoothly.  Definitely share some images and tests to show how much performance you gain :-)

hehe, well it's an interesting puzzle. The main issue so far that I'm running into (and if I'd of only Googled it I'd know) is NVME and legacy bios mobos don't behave well together when it comes to making them bootable. Looking into USB boot loaders now -- Clover and Duet. Both of which have the most convoluted and confusing instructions ever. They exist, so I know it's possible. It's just that these instructions are written in such a manner as to give me nerd rage. So far 5 or 6 how-to's are most definitely not working as per instructions. I believe mostly due to them linking to sites for the boot loaders that have kept on being developed...making the old instructions...well...old (and inaccurate) 

Brad_Hawthorne
hehe, well it's an interesting puzzle. The main issue so far that I'm running into (and if I'd of only Googled it I'd know) is NVME and legacy bios mobos don't behave well together when it comes to making them bootable. Looking into USB boot loaders now -- Clover and Duet. Both of which have the most convoluted and confusing instructions ever. They exist, so I know it's possible. It's just that these instructions are written in such a manner as to give me nerd rage. So far 5 or 6 how-to's are most definitely not working as per instructions. I believe mostly due to them linking to sites for the boot loaders that have kept on being developed...making the old instructions...well...old (and inaccurate) 

I'm curious if you get this to work. 

I didn't use 970 Evo Plus with adapter card as boot drive in my X58 but I did use it briefly as data drive and in average daily workflow there was no perceived practical performance difference it and one sitting in system with PCIe 3.0. Like mentioned, it is all about workload and if your app is not creating enough I/O to saturate that bus for long time you won't be realizing any performance differences worth thinking about.

After doing some testing I'm pondering a combination of things on the X58. My machine has 48GB of DDR3 in it. I've allocated 32GB of it to a Primo Ramdisk, which leaves 16GB of ram for Windows use. Pondering just loading the OS into the ramdisk and running everything else on the NVME drive. The bottleneck on the X58 is drive I/O. Getting up to 850MB/s I/O on the NVME while the Ramdisk is getting up to 5,500MB/s reads and 12,000MB/s writes (Yes, the write speed is more than the reads for some reason). In either case, both are very responsive in real world use compared to legacy spindle drives. I'm still working through the boot issue and am looking into boot options for both drives at the moment.

Brad_Hawthorne
The bottleneck on the X58 is drive I/O. Getting up to 850MB/s I/O on the NVME while the Ramdisk is getting up to 5,500MB/s reads and 12,000MB/s writes ...

Whether you will be able to have any real world benefit from such split is strictly dependent on type of workload distribution you will have. You will know best. In any case, figures you are getting from NVME seem much lower than they should be. I don't remember exact figures I was getting when I tried same so my memory might be playing games on me but I am under impression I was getting close to supposed bandwidth of PCIe 2.0 4 lanes. Try increasing queue depth and number of threads to make sure CPU is not your bottleneck. If that doesn't help check is your adapter card in slot that has 4 lanes and, if yes, that adapter card itself is not an issue.
 
P.S. I was under impression EVGA X58 supports only 24GB?