Great review.  I hope that you get the tri-SLI issues worked out.

I wonder if the fan in gtx 670 FTW noisy in load?

r-trus

I wonder if the fan in gtx 670 FTW noisy in load?

 
The cards are very cool and quiet when ran with room to breathe or not stacked together. The fans get very loud and they have a hard time staying cool when they are bricked together in 3-way SLI. That's why I'm currently installing water blocks .

how my 3dmark score for my little babies http://3dmark.com/3dm11/3647046

raw2dogmeat
You don't have enough vram. Period. Not to run 2x or 4x msaa. I have 2 x 4gb SC's and personally see 1800+ with all setting maxed and NO msaa. Turn on 2x msaa and the vram jumps to 2100 mb+. 4x msaa uses 2400mb+. No joke. This is however on multiplayer. I have'nt had time to check single player results.

 
I fixed the crashing issue. I re-installed Nvidia drivers and put cpu back to stock and then a light OC @ 4.2ghz (dont know which one was causing the issue). 
 
BF3 Ultra settings 5760x1080 on 64 multiplayer max VRAM with Aero disabled is:
FXAA - 1700mb
2xMSAA- 1950mb
4xMSAA - ( get DX11 error)
 
Campaign ultra settings  5760x1080 Aero disabled
FXAA- 1386mb max
2xMSAA- 1500mb
4xMSAA- maxes VRAM 1825mb with no issues.

Heres my stock scores running 3dmark11
http://3dmark.com/3dm11/3642692

Swolern

Thebski do you have any issues with low GPU usage on BF3 online multiplayer with 3-way SLI?

With my 4-way SLI my GTX 670s usage only ranges from 55-75%, and this is 5760x1080 @ 120hz with Vsync off. I dont see a bottleneck anywhere with CPU usage maxes @ 75-80% and ram @ 45%. Also bf3 campaign with same settings uses 95-100% GPU usage and I get a much higher framerate. I'm thinking this is a driver related issue.


BTW BF3 Close Quarters maps are the $hit!!! Fun as hell!

 
Yea my GPU usages in either SLI or 3-way SLI are odd in BF3 online. They bounce around a lot. They are higher in SLI mostly staying above 80% on both GPU's, but in 3-way SLI they occasionally dip as low as 50%. Not sure what's going on with that. I'm hoping it's just driver related. It's not my CPU as my CPU usage maxes out at about 50% and during the times that the GPU usages dip so does my CPU usage. It will dip to around 22% about the same time my cards dip somewhere in the 50%-70% range.
 
I should note the usages are just all over the place, meaning they aren't all low together or high together.. I might be seeing 95% usage on one, 75% on the other, and 80% on the 3rd or something like that. Just funky stuff. Hoping the next driver will be out soon and will address some of that.
 
I know in my case it is not a memory capacity bottleneck. I'm seeing these funny usages even only running FXAA when I have 200-300 MB of memory left. It's too bad because the few times I do see them all running 95%+ the frame rates are just unreal. It would constantly be at 100+ FPS if these cards could maintain 95%+ GPU usage across the 3. I'm usually in the 70's and 80's for frame rate, but it's not uncommon for me to catch it at 120 FPS or even 130 FPS if I can catch all cards working at 99%. These things are really, really fast.

I also had a thought that maybe it was heat related because the top two cards do have a hard time staying below 70C when they are all bricked together in 3-way SLI. I will know the answer to that soon. The waterblocks are currently installed on the cards, but I'm waiting on the EK FC Link parts to make the connection to EK's SLI bridge. I didn't know the links weren't included with the bridge until I went to put everything together 

thebski
 
 We don't see scaling quite as good here as we did with the graphics score on 3DMark, and something is obviously wrong with this benchmark in surround resolutions and SLI configurations. It doesn't concern me too much because these benches don't mean too much to me, but this is what I found.

 
 
I've been away from these forums for a while, but I'm pretty sure some of your performancce issues are related to PCI-E bandwidth.  Although I don't have a link handy, I've seen testing by other users (http://www.xtremesystems.org/forums/showthread.php?279308-Vega-s-*Heavyweight*-display-and-computer-edition-2012&p=5081386&viewfull=1#post5081386) via hardforum regarding gtx680 3/4 way sli on pci-e 2.0/3.0.  When hacking pci-e 3.0 support onto x79 systems, performance and scaling improved dramatically.  
 
Now, I only have 2 gtx680's so I can't speak first hand for 3 way, but I do know that with heaven 3.0, I get almost 100% 2 way sli scaling @ 5760x1080.  However, my x58 board is running both cards at x16, whereas your z68 board at best runs x16/x8.  To really open up those gpu's, you may want to consider upgrading to Ivy Bridge/Z77 (or in the above linked case, x79 with hacked drivers)

crazyrob

thebski
 
We don't see scaling quite as good here as we did with the graphics score on 3DMark, and something is obviously wrong with this benchmark in surround resolutions and SLI configurations. It doesn't concern me too much because these benches don't mean too much to me, but this is what I found.

 

I've been away from these forums for a while, but I'm pretty sure some of your performancce issues are related to PCI-E bandwidth.  Although I don't have a link handy, I've seen testing by other users (http://www.xtremesystems.org/forums/showthread.php?279308-Vega-s-*Heavyweight*-display-and-computer-edition-2012&p=5081386&viewfull=1#post5081386) via hardforum regarding gtx680 3/4 way sli on pci-e 2.0/3.0.  When hacking pci-e 3.0 support onto x79 systems, performance and scaling improved dramatically.  

Now, I only have 2 gtx680's so I can't speak first hand for 3 way, but I do know that with heaven 3.0, I get almost 100% 2 way sli scaling @ 5760x1080.  However, my x58 board is running both cards at x16, whereas your z68 board at best runs x16/x8.  To really open up those gpu's, you may want to consider upgrading to Ivy Bridge/Z77 (or in the above linked case, x79 with hacked drivers)

 
I think you're probably right. I've long suspected a bottleneck somewhere with my erratic GPU usages, and PCI-E is the next place I could see bottlenecking. Problem is I've spent over $5000 on this computer in the last 4 months and closed on my first house 2 weeks ago so I need to take a spending break for a few months.
 
I knew when I bought Z68 in January I should have bought X79, but I seriously didn't think I'd ever be running over 2-way SLI. I don't know how much Ivy Bridge interests me although Z77 is a nice platform, so if I were to switch I'd switch to X79 to have a ton of PCI bandwidth. Have they fixed the issue with X79 and PCI-E 3.0 yet? I thought I was reading that sometimes the registry hack doesn't work or something like that.
 
Edit: 
 
After reading Vegas review, I have no doubt that my problems are from PCI-E bandwidth. His description about GPU usages falling in the 50% range like he's CPU limited fit what my system is doing exactly.
 
Ugh, time to save for X79 now. PCI-E 3.0 doesn't work on X79 without the hack does it? And is the hack a reliable way to run PCI-E 3.0 all the time? I don't want to invest another $500 to upgrade to X79 if I won't be able to run PCI-E 3.0 all the time.

ikeyes

I will be testing on a new board as well, since my P8Z77-V Deluxe is "limited" to only PCI-E 3.0 x8/x8 SLI
I will be testing everying at 5760x1080, no bezel correction, so it's easily comparable with others.  I will be using 301.42 WHQL's from Geforce.com.


 
From everything I've read PCI-e 3.0x8 is just fine for these cards as you get bandwidth equivalent to PCI-e 2.0x16. I'll try to find the review I read that in. I honestly believe that the OP's biggest limiting factor for the tri-SLI and even the SLI is the lack of PCI-e bandwidth. I know 3dmark11 isn't an end all be all benchmark but the fact I posted a scored on par with him with a 3 year old platform is telling to me.
 
EDIT:  Found it: http://www.techpowerup.com/reviews/Intel/Ivy_Bridge_PCI-Express_Scaling/
Kind of hard to read their graphs but great info.
EDIT2: My 3DMark11 score for reference: http://3dmark.com/3dm11/3626071



ikeyes
I picked up a Gigabyte Z77 G1 Sniper 3 to remedy that and regardless of my findings I expect I'll keep it... I've only just got it out of the box and it's the nicest board I've ever layed my eyes on.  It will allow me to run PCI-E 3.0 x16/x16 SLI and x16/x8/x16 Tri-SLI and maybe someday Quad-SLI.

Last night I just popped in two 670 4GB's into my current setup, doing a clean install of the 301.41 WHQL's I have been using for the last few weeks.

I'll get more detailed results up in the next few days once I have everything gathered, but I'll just say that for BF3 even with 4GB there is absolutely no difference in performance if even it's a bit less with the lower clock speeds.

With 4xMSAA enabled I am getting the same FPS or a tad less as with my 670 FTW SLI and it is using about 2400MB of VRAM but that's not the problem as I suspected, it's that these GPU's cannot push out that high of frames at 5760x1080... so I can't wait to test this again on the new board and then introduce the 3rd GPU.

I still feel that raw GPU horsepower is going to be the limiting factor for these first generation Kepler cards...  but we'll let the numbers speak for themselves.

I look forward to your results. I also agree that the GPU is getting overwhelmed before VRAM is truly becoming an issue with these cards. Maybe by the 700 series cards 4GB will be a norm/truly useful.

ikeyes

I agree that PCI-E 3.0 x8 should be enough, I really updated the board because the Deluxe was two-way SLI only.   Other than that I would keep it, even with it's quirks it's been awesome.

However the G1 Sniper 3 was $5.00 more so... 

For $5 why not?
 
I'm excited to put together my X79 that's waiting at home for me. I'm wondering how the 3930k I got will stack up against my current 970. To keep from going on a tangent this thread is still full of win, I hope someone will post some in depth 4GB card numbers using the same data types as the OP did with his 2GB cards.
 
On an aside my BF3 was only running at about 50% GPU usage until I flipped the pwr mgmt mode from adaptive to perfer max performance. My FPS is higher but the cards still only hit about 70-80 usage. I wonder if it is driver related?

TChittenden II
On an aside my BF3 was only running at about 50% GPU usage until I flipped the pwr mgmt mode from adaptive to perfer max performance. My FPS is higher but the cards still only hit about 70-80 usage. I wonder if it is driver related?

 
My 670 FTW or 670 4GB cards in SLI are 97-99% GPU usage in everything I've tested so far, BF3, Crysis 2, Heaven, AvP DX11 Bench... you should absolutely be seeing high GPU utilization with two 670's in SLI even at lower resolutions providing you don't frame cap the games ie. V-Sync, Target Frame Rate, etc
 
What resolution are you using?

@thebski I wish I could say x79, but from forum users results being so hit or miss with that registry/driver hack, it's a tough call.  Especially since no one knows if this mod will continue to work in future driver revisions if Nvidia doesn't get it sorted out.  ATM, since gaming rarely benefits from a 6 core cpu, I'd go z77 with a board like P8Z77 WS or P8Z77-V Premium, both of which include PLX chips for extra pci-e bandwidth.  Or maybe EVGA's new Z77 FTW board, although I haven't seen any reviews of it yet.

ikeyes,
 
I'm game to do whatever 3-way 670 FTW testing necessary to match your 3-way 670 4GB. No worries on the time frame either.
 
As far as graphics cards I'm pretty much set that I'm keeping these. I'm pretty sure my crash issues have to do with either my USB ports on my mobo, a device I'm connecting to via USB, or a slim chance it's my Blu Ray drive since I just painted it Matte Black to match everything else in the comp, but I taped it up really well and screwed off all holes so the chances of any paint reaching the internals is slim to none.
 
As far as memory goes, I think you may be right that 2GB is enough to satisfy the GPU horsepower these have unless you just have to run huge MSAA. I don't necessarily have to have MSAA if a game doesn't allow it memory wise, and I'm much more concerned with keeping frames above 60 so the 3-way is nice.
 
In fact, I'm leak testing my new loop with the waterblocks on the GPU's as we speak , so I'll only be able to do 3-way testing from here on out.
 
As far as X79 vs Z77, it is a tough call and I hate tough decisions lol. If I were guaranteed to be able to us PCI 3.0 on X79 there would be no question, but I'd really hate to spend even more money than Z77 and not have what I even upgraded for. 
 
Edit:
 
If I did go Z77, I'd have to wait for the ROG boards from Asus. I'm dead stuck on the red and black theme. The entire PC was built on that theme and I can't pull on a fast one on the mobo at this point. The Maximus V would be nice since I would have enough PCI slots for my 3 cards and my Sound Blaster card.

thebski
If I did go Z77, I'd have to wait for the ROG boards from Asus. I'm dead stuck on the red and black theme. The entire PC was built on that theme and I can't pull on a fast one on the mobo at this point. The Maximus V would be nice since I would have enough PCI slots for my 3 cards and my Sound Blaster card.

 
The Z77 FTW doesn't do it for you?  Red & Black to the max man  hehe  I've had three EVGA boards over the years, not an issue with any of them.  I can't even remember the first board from EVGA I got, but it was for my AMD 939 4800+ / 7800GT SLI setup that I got specifically to play BF2!