HeavyHemi
 
Something else I'm not sure you're aware of. When dedicating PhysX to CPU in the Nvidia Control Panel it sets the PhysX setting in game to normal. You can't set it to high. It will stay on normal until you set PhysX back to either auto or in some manner using a GPU in the Nvidia Control Panel. Then you have to reset the PhysX to High in the game options and restart the game. I bring this up because your SLI TITAN FPS look like that might have been run with PhysX on normal instead of high. Also it reveals, that since the game forces PhysX to normal when using the CPU, the performance for even reduced PhysX on the CPU is terrible as you indicated.

Thanks!  I will check this later tonight when I get back to my home office.  Though, I'm pretty sure I checked the settings on the last run I made (Titan standalone) and it was still set to HIGH.  But I'll check it with CPU and all and such and will report back.
 
If you compare the Titan StandAlone to the Titan SLI, they are almost identical.  The roughly 15% increase in going from a single GPU to 2-way SLI GPUs matches what others have posted online.  So, I'd say it was an accurate test.
 
Lastly, each "run" was at least 2 runs in each configuration ran back to back.  And then a reboot when the configuration changed.  I mention this because on some runs when there was a significant difference between the back to back runs, I ran it 3 or 4 times with reboots until I got an accurate test.  For example, the Titan w/CPU PhysX was all over the place.  I found out that if I closed Google Hangout, it was far more reliable.  But, it me like 6 runs until I got a consistent back-to-back run (and the one posted above is actually the best out of all runs).
 
But I'll verify and will post back.
 

HeavyHemiThis line is a bit confusing as related to your graphs: "SLI was always disabled when using the 2x Titans.  I used the PhysX Settings to designated either Titan GPU1 or Titan GPU2 as the PhysX engine, with GPU2 checked off as "Dedicated to PhysX."  Did you run with the TITANS in SLI with PhysX on auto with just the two TITANS installed?

I can see how that is a bit confusing now.  Thanks again.  I'll try to re-word it in a bit.
 
When I first wrote that line, I had not made a Titan 2-Way SLI run.  All tests were purely a single Titan, plus whatever I was messing with for the PhysX (a 2nd Titan vs GTX460 vs CPU vs None).  That last one, "None", is what I meant by manually setting it to GPU1.
 

 
 
When I did the Titan w/Titan PhysX runs, I manually designated GPU2 as the PhysX card and checked the box for "Dedicated to PhysX" - there was no SLI enabled.  
 

 
That checkbox, "Dedicated to PhysX" is not available when you are using the primary GPU as PhysX manually set.  You can only check it on other GPUs.
 
 
And just for completion of all settings:
 

 
I did not just take these screenshots.  I took each screenshot, including in the OP, when I made the change.  I also rebooted after each change to ensure no funkiness with the Nvidia drivers.  This is why some monitors are flipped around as I was tring to get Surround working earlier on a single GPU without success when I had two identical GPUs in the system.
 
So, it is confusing because I was trying to encompass all of that information in a single sentence.  But you are right, there are a lot of specifics to each test run.
 
The ONLY time I left the PhysX set to "Auto" was for the Titan 2-way SLI runs.  All other times, I manually set it (and even captured screenshots of each setting).  Sadly, I didn't grab a screenshot of that setting as, well, that's the default when you enable SLI anyways - set to Auto, and it designated GPU2 as the PhysX engine.
 
 

HeavyHemiSomething else I'm not sure you're aware of. When dedicating PhysX to CPU in the Nvidia Control Panel it sets the PhysX setting in game to normal. You can't set it to high. It will stay on normal until you set PhysX back to either auto or in some manner using a GPU in the Nvidia Control Panel. Then you have to reset the PhysX to High in the game options and restart the game. I bring this up because your SLI TITAN FPS look like that might have been run with PhysX on normal instead of high. Also it reveals, that since the game forces PhysX to normal when using the CPU, the performance for even reduced PhysX on the CPU is terrible as you indicated. 

If this is true, then this may account why the Titan + CPU benchmark seems really high at times yes.  But, it would contradict why it was so low for two big scenes.  Unless, as I stated in the OP, the CPU as an Nvidia PhysX engine really is that poor when using a lot of PhysX.  Even set to Normal really brings down the FPS.  
 
But agian, I'll check when I get a chance to see if it lowers.  I was using the latest patches as of Sunday, December 15th.

IMO; what a waste! :-) Where is your Tri-SLI benchmark?

shadowboricua
IMO; what a waste! :-) Where is your Tri-SLI benchmark?

PM me for my address and you can send a 3rd Titan my way.  I'll pay for shipping to ship it back!
 
But yeah, I did feel this was a little bit of a waste not being at 5760x1080.
 
Been thinking of making some additional runs now at 5760x1080 with the following:

• 2-way SLI Titans w/PhysX set to one of the Titans
• 2-way SLI Titans w/GTX 460 as PhysX
• 2-way SLI Titans w/CPU as PhysX

 
That should nail it for me, though I don't have a 3rd Titan for that PhysX score.  The GTX 460 sammiched between the Titans would be a problem since I have ACX cooler currently on the Titans.  I'm about to switch to reference coolers when I get back from the holidays because the heat output in the chassis is just unreal and increases temps far too much (have to keep the side off for normal temps).
 
I do have an 780 Classified sitting in a box as well to see how much better PhysX would be compared to that GTX 460.  But, we already saw in the first test in the OP that a faster GPU really does affect PhysX, even at that low resolution!  The problem with the 780 Classified is that it is too an ACX cooler.  
 
I have it on step-up to an 780 Ti.  Was going to sell that 780 Ti new-in-box to make up the cost of the used Titan I got to replace it; but, if I can afford it, I just may keep it now as a PhysX card.  Wow, what a waste that sounds like...  LOL
 
But, with 3D Surround Vision enabled at 5760x1080 - you need ever FPS you can get because 3D Vision cuts the frames in half, and I'm already struggling with 2-way Titans at that resolution.  In the words of my wife playing Batman Origins with her 3D Vision glasses on surrounded by tri-monitors: "I feel so dizzy, and scared.  Everyone is trying to kill me!" (and she jumps when someone swings from the side)

I was so impressed with your analysis that I went out and bought an EVGA Titan Superclocked today.  PhysX is hugely important and your results clearly show that.  If the black edition comes out than I can use my card as a PhysX down the road.  Thanks again for running the test.

@HeavyHemi: Looks like you were right.  When selecting CPU as the PhysX, it drops PhysX in the game down to Normal.  Ok, no problem and I stand behind that benchmark as it won't change.
 
But, a number of other tests were run after that - all of which were set to Normal after that CPU-flux-up whereas they should have been set to HIGH.  Off the top of my head, here are the ones that will need to be re-ran:

• Titan SLI
• Titan w/Titan PhysX with Double Precision enabled.
• Titan Standalone

I estimate lower FPS in the above due to heavier PhysX next time around.  
 
But since PhysX cripples the FPS when running on the same GPU (or in SLI), the GPU usage doesn't do much anyways.
 

eduncan911
@HeavyHemi: Looks like you were right.  When selecting CPU as the PhysX, it drops PhysX in the game down to Normal.  Ok, no problem and I stand behind that benchmark as it won't change.
 
But, a number of other tests were run after that - all of which were set to Normal after that CPU-flux-up whereas they should have been set to HIGH.  Off the top of my head, here are the ones that will need to be re-ran:

• Titan SLI
• Titan w/Titan PhysX with Double Precision enabled.
• Titan Standalone

I estimate lower FPS in the above due to heavier PhysX next time around.  
 
But since PhysX cripples the FPS when running on the same GPU (or in SLI), the GPU usage doesn't do much anyways.
 

I kinda figured that as I'm running SLI TITANS and I knew about what the FPS should be. Based on my experience, setting the game profile to single GPU in the Nvidia control panel (leave SLI enabled) with PhysX on auto seems to work pretty well and performs slightly better than having SLI enabled. Using this method doesn't affect any other games that use SLI.

HeavyHemi
 
 
I kinda figured that as I'm running SLI TITANS and I knew about what the FPS should be. Based on my experience, setting the game profile to single GPU in the Nvidia control panel (leave SLI enabled) with PhysX on auto seems to work pretty well and performs slightly better than having SLI enabled. Using this method doesn't affect any other games that use SLI.

You basically confirmed exactly what the result was from my tests: a single Titan + 2nd Titan as PhysX is better than 2-way Titans SLI at 1920x1080.  I bet your "Auto" setting selected the 2nd GPU, the one that isn't being used.
 
I don't have luxury with 5760x1080 surround though (SLi required to keep FPS up).  
 
When I get back from the holidays, I'll slap the reference coolers back and and throw in the gtx 460 and make some 5760x1080 runs.

FalconX79Dark
I was so impressed with your analysis that I went out and bought an EVGA Titan Superclocked today.  PhysX is hugely important and your results clearly show that.  If the black edition comes out than I can use my card as a PhysX down the road.  Thanks again for running the test.

Wow, now that was some coin! if the black edition/titan ultra/Titan 2 or something comes out in the 90 days, you may be able to Step Up - if EVGA puts it on their step up list.

Currently though, the normal Titan is not on the step up - which makes it a low chance that the premium priced Titan 2 will not be either.

http://www.evga.com/support/stepup/

FalconX79Dark
I was so impressed with your analysis that I went out and bought an EVGA Titan Superclocked today.  PhysX is hugely important and your results clearly show that.  If the black edition comes out than I can use my card as a PhysX down the road.  Thanks again for running the test.

PhysX is hugely important even though a majority of games don't use PhysX?